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recommendations for changes to the EBQ. A content analysis was also conducted 
of published research papers that have involved analyses of the EBQ data to 
determine how the data were used in these research analyses . The study also 
drew upon recommendations garnered by a series of focus groups conducted by 
the American Institutes for Research in 1998. These sources were integrated 
with the study's own analysis of the NALS data. The following changes to the 
EBQ for future adult literacy assessments were recommended: expand the 
questions pertaining to educational experiences; expand the section on social 
and political participation or move the current items to other areas and 
delete this section; add an item on voter registration; acquire more detailed 
information regarding work history, wages, and parents' occupations; expand 
the section pertaining to literacy practices; and gather information on 
technological literacy practices. (Appendixes include 34 references; 
instruments; and studies examined for content review.) ( YLB) 
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Abstract 



This report describes how social science researchers and other experts have analyzed data from the 
National Adult Literacy Survey (NALS) since the public release of the data in 1993 to serve a variety of 
educational, social policy and scientific purposes. The particular focus of our report pertains to researchers’ 
uses of the English Background Questionnaire (EBQ) portion of the NALS. Our data were gathered by 
surveying NALS data users about how they have used the EBQ data, their perspectives on the strengths and 
weaknesses of the EBQ, and their recommendations for changes to the EBQ. We also conducted a content 
analysis of published research papers that have involved analyses of the EBQ data to determine how the 
data were used in these research analyses. Finally, we drew upon the recommendations garnered by a series 
of focus groups conducted by the American Institutes for Research in 1998. After integrating these sources 
with our own analyses of the NALS data, we recommend several changes to the EBQ for future adult 
literacy assessments. These are as follows: (1) expand the questions pertaining to educational experiences 
including adult basic education and other nonformal types of educational experiences; (2) expand the 
section on social and political participation or move the current items to other areas and delete this section; 

(3) add an item on voter registration; (4) acquire more detailed information regarding work history, wages 
and parents’ occupations; (5) expand the section pertaining to literacy practices; and (6) gather information 
on technological literacy practices. We also recommend the following actions: (1) re-examine the issue of 
the unidimensionality of the literacy scales; (2) utilize Reder’s Lifelong Learning Questionnaire for new 
items; (3) link the NAAL to the Adult Education Survey of the National Household Education Survey; and 

(4) collect data at multiple levels of the survey design. 
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The data for the National Adult Literacy Survey (NALS) were gathered over the first eight months 
of 1 992. The first report summarizing the results of this largest-ever assessment of adult literacy skills was 
issued in September, 1993. The data were subsequently released in the public domain in 1994 by the 
National Center for Education Statistics (NCES). The data were therefore available for educational 
policymakers and planners; researchers in universities and research organizations; and experts in various 
fields such as assessment, labor economics, adult education, sociology, and political science to utilize for 
the purposes of secondary analyses. By making the data available to experts for analysis, additional 
findings, insights, and implications for policy, program development, and practice can be generated from 
the NALS and disseminated far beyond the capabilities of a single agency such as NCES. 

Large-scale social science data, of the type obtained in the NALS, are useful for addressing a 
number of issues of interest to educators and social science researchers. 

For example: 

1 . The scope and extent of adult literacy; 

2. The ways in which literacy contributes to individuals’ well-being, social involvement, and 
workplace performance; 

3. The associations of various demographic variables such as age, gender, and educational 
attainment to literacy proficiencies; and, 

4. The nature of observed differences in literacy skills and practices among the different racial, 
ethnic, age, and geographic groups throughout the United States. 

The purpose of this report is to describe how social science researchers, educators, and policy 
experts have used the NALS data since the public release of the data in 1993 to serve a variety of 
educational, social policy and scientific purposes. The particular focus of this report pertains to the manner 
in which secondary analyses of the data have been conducted, particularly as these analyses have employed 
data from the NALS English-language Background Questionnaire (EBQ). Several issues are of interest to 
us, and we have attempted to address these in our investigation. We gathered information in two ways: (1) 
via a survey of identified NALS data users; and (2) through a content analysis of the published research 
papers pertaining to the NALS in which EBQ data were used. 
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We began by conducting a limited survey of several key personnel with expertise in either all or 
specific sections of the EBQ. An e-mail survey was sent to identified NALS data users to ascertain their 
perspectives on the adequacy of the data provided by the EBQ. The goal was to ascertain individuals’ 
understandings and uses of these data and their recommendations — for gathering background data — based 
on their experiences using the data derived from the NALS background questionnaire. 

The key issues resulting from our survey responses are reported here and integrated with 
information obtained from our content review of a small number of published studies (i.e., secondary 
analyses) of the NALS. The focus of our content review is to identify both the specific background 
variables used, and how these variables and analyses appeared to have either contributed to or limited the 
researchers’ inferences and/or policy recommendations. Our assumption is that such information is 
important to the National Assessment of Adult Literacy (NAAL) survey designers and will ensure that 
critical questions are considered, which might then enhance researchers’ and policymakers’ abilities to draw 
meaningful conclusions and/or make policy recommendations. 

We also weigh the relative advantages and disadvantages of adding items and modifying existing 
items on the EBQ. Any changes must, of course, be considered in light of the potential costs — in terms of 
real dollars and the sacrifice of useful data. Changing the EBQ may, for example, impede comparisons and 
trend analyses for the 1992 and 2002 adult literacy assessments. While it may be possible to create a 
parsimonious set of background questionnaire items that provide essential information and allow strong 
inferences about adult literacy, this must be accomplished within the parameters of minimal administrative 
and data management costs and advantages to data reporting. 

The 1992 NALS English-language Background Questionnaire contained six “content areas” 
enabling the collection of information on survey participants’: (1) general and language background; (2) 
educational attainment and experiences; (3) political and social participation; (4) labor force participation; 
(5) literacy activities and collaboration with others, and, (6) demographic information. These data were 
deemed by the NALS developers to be important for the purpose of “[gaining] an understanding of the ways 
in which personal characteristics are associated with demonstrated performance” on the three NALS literacy 
proficiency scales — prose, document, and quantitative literacy (Kirsch, Jungeblut, Jenkins, and Kolstad, 
1993, 5). Adult literacy researchers and policy analysts vary in their areas of expertise in relation to these 
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components of the EBQ, and they utilize the data in different ways in order to address questions of interest 
to themselves and/or to their organizations, institutions, or sponsors. Thus, we attempted to obtain a diverse 
sample of NALS data users and analysts. 

Identification of NALS Data Users 

NCES has a NALS Web site that contains a link to a list of NALS “data users” 
(http://nces.ed.gov/naal/naal92/users.html). This list, dated May 16, 1996, consists of 70 names of 
individuals (including the first author of this report) who are academic researchers, policymakers and 
analysts for public and private research firms and agencies, journalists, and other experts. The listed 
individuals are those who had requested a copy of the data files on diskette from NCES. Because 
individuals were able to download the data directly from the Web site as of mid- 1996, it was not possible to 
maintain a more up-to-date file of data users (A. Kolstad — personal communication, May, 1998). We 
attempted to contact as many individuals on this list as possible to gather information regarding their uses 
and analyses of the NALS data. Our primary method for contacting individuals was through e-mail 
(addresses were available for 58 persons on the list, 83 percent). Several of these addresses were invalid, 
however, so we then attempted to obtain up-to-date e-mail addresses and phone numbers. 

An additional, updated, list of 1 1 names of NALS data users was also provided to us through 
Andrew Kolstad at NCES. We received replies from two persons among this subgroup, one of whom 
reported not having used the data for statistical analyses due to its complexity. A few of the individuals that 
we surveyed also referred us to other researchers with whom they had collaborated on their NALS studies. 
We were then able to complete one additional telephone interview. We attempted to contact, via e-mail, the 
state-level directors of adult education in each of the states that had conducted NALS state-level 
assessments. Although we did not expect that this latter group would have conducted analyses of the NALS 
data, they could presumably refer us to statisticians and policy analysts on their staffs who had done so. Our 
total population of identified “NALS users” equaled 94 persons. 

Because the authors of this report have conducted analyses of the NALS data, resulting in one 
published research paper, another research report that is now in press, and five paper presentations at 
national conferences, we are familiar with several other researchers who have analyzed the NALS data. The 
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first author of this report has also edited a book on the NALS that contains seven chapters by individual 
investigators that describe secondary analyses of the NALS data. Also, the first author has reviewed other 
research papers based on the NALS. Therefore, the authors of this report are reasonably well informed 
about the body of scholarship based on the NALS that has been produced over the past five years. However, 
because it would be presumptuous to assume that we are aware of all research pertaining to the NALS, we 
conducted an electronic database search of educational and social science literature to determine and locate 
all published research papers based on the NALS. The electronic search revealed two additional empirical 
studies employing the NALS data — one of which used the EBQ variables. 

Finally, we sent a request to the National Literacy Advocacy (NLA) electronic listserv in order to 
notify NALS data users whom we may have overlooked. The purpose of the NLA listserv is to provide 
information about issues affecting federal legislation, policy, and funding for literacy, basic and secondary 
education and English for Speakers of Other Languages. Thus, it is a particularly useful resource for 
policymakers and program providers in the adult literacy field. Our message to the NLA listserv read as 
follows: 



I am preparing, along with my colleague Janet Sheehan-Holt, a report pertaining to the 
various uses of the 1 992 National Adult Literacy Survey by adult literacy experts — particularly in 
regards to the Background Survey (i.e., language background of respondents, educational 
background and experiences, labor force and social participation, literacy activities, demographic 
information). The purpose of the report is to recommend changes in the Background Survey for the 
planned National Assessment of Adult Literacy (NAAL) in 2002. We are particularly interested in 
any secondary descriptive or inferential analyses of the NALS as conducted by policy experts at 
the federal and state levels, academic researchers, and others. 

Although we have identified a number of individuals who have conducted, and 
subsequently published such analyses, we want to make sure we haven’t overlooked anyone. If you 
have used the NALS Background Survey data for secondary analysis, we would like to include you 
in our sample. Please contact us by email (mcsmith@niu.edu) or telephone (815) 753-8448 for 
more complete information. We have a very brief survey that can be completed via email. 



Unfortunately, we received only one response to this message. We re-posted our request to the NLA listserv 
on May 1 st , 1999, to further canvass NALS data users, but received no responses. We believe that this lack 
of response is largely due to the fact that many NLA subscribers are directors of adult literacy programs 
rather than researchers. As such, these individuals are most likely to refer to the NALS results — rather than 
to conduct analyses of the data — to educate their respective communities about the need for adult literacy 
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education, and to justify their funding requests in order to raise funds, increase services, and improve 
service delivery. 



Survey 



Our initial e-mail query (see Attachment A) identified the purposes of our project, and asked 
individuals to respond briefly to three questions, including their willingness to complete a second brief 
survey via e-mail or by telephone. Through this method, we received replies from about one-third of the 
identified NALS users (n=30). Sixteen of these persons indicated having used the NALS data for secondary 
analyses (in addition, two persons used state-level data and a third used NALS data to replace missing data 
for the International Adult Literacy Survey; the two authors of this report have, of course, also used the 
NALS for secondary analyses) and nine had not conducted any data analyses. All of those who responded to 
our initial survey, and who have analyzed the NALS data, agreed to respond to the second part of our 
survey. 

The purpose of the second part was to determine (1) the specific sections of the EBQ of greatest 
use to these researchers, (2) the specific questions of interest to their research, and (3) their perspectives on 
the relative strengths and weaknesses of the EBQ for making strong inferences from the data to populations 
and subgroups of interest to them. Upon affirmation that a respondent would complete the second part of 
the survey (see Attachment B), it was sent via e-mail to those individuals — with two exceptions. These 
persons were either interviewed over the phone or face-to-face. 

Published Research Reports 

We identified more than a dozen research papers that have been published in the scientific 
literature (i.e., journals), or are currently in press, or are contained in the aforementioned edited book 
(Smith 1998). All of these reports are based upon secondary analyses of the NALS data (see Attachment C). 
Three of these papers involve primarily descriptive analyses examining particular populations (e.g., 
community college students). The remaining reports are largely based on inferential statistical analyses. 

Two book chapters (see Finn and Gerber 1998; Howard and Obetz 1998) are essentially summaries of 
studies published as journal articles (Gerber and Finn 1998; Howard and Obetz 1996). In these cases, we 
examined the journal articles. In another case, a chapter summarized a previously published study (see 
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Smith 1996), but also reported the results of a second analysis that had not been previously reported (see 
Smith and Sheehan 1998). Also, Pryor and Schaffer have reported utilizing or referencing the NALS data 
throughout their book (Pryor and Schaffer 1999) to support their claims pertaining to associations between 
education and literacy, and employment and earnings. The NALS data were not the primary data source for 
their book, however. We examined a research report by Pryor and Schaffer (1997) in which they conducted 
secondary analyses of the NALS data. 

Unpublished NALS Reports 

Our database search also revealed that there are an additional fifty-five reports pertaining to the 
NALS that are available through the Educational Resources Information Clearinghouse (ERIC) document 
retrieval service. These ERIC documents include: the original executive summary of the NALS (Kirsch et 
al. 1993); various state-level reports prepared by the Educational Testing Service; state, local and national 
estimates of literacy; literacy performance of various populations (e.g., older adults, prisoners); essays on 
the state of adult literacy; and, methodology and technical papers pertaining to adult literacy assessment. 
These reports were not used in our analyses; however, as they involved descriptive reporting of data directly 
from the NALS results and/or the NALS Executive Summary, rather than original analyses of the NALS 
and, in particular, the EBQ data. 

Researchers * Uses of the EBQ Data 

Because our analysis of uses of the EBQ data is largely a qualitative one, we will refer to 
respondents to our survey as “informants” in order to avoid confusion between NALS sample respondents 
and those persons who completed our survey. Slightly more than one-third of the informants (11 of 30) from 
our initial survey indicated the specific sections of the EBQ that they utilized in their data analyses (e.g., 
general and language background, educational background, demographic information). As might be 
expected, there was slightly greater focus on the items pertaining to NALS respondents’ educational 
background and experiences. However, five researchers reported having used data from all sections of the 
EBQ; others’ analyses have been limited to two, three, or four sections only (e.g., political and social 
participation, literacy activities and collaboration), depending upon the particular questions posed in their 
research. 
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The second part of our e-mail survey asked informants to identify the research questions of interest 
to their studies. Further, we asked that they indicate how the EBQ items strengthened their ability to draw 
inferences from the data, how the items may have limited such inferences, and what items or sections should 
be revised or dropped for the NAAL. Eleven individuals completed our second survey via e-mail or were 
interviewed in person or on the telephone. 

Survey Results 

A few individuals responded to our initial inquiry and identified themselves as employees of state 
divisions or adult education programs; a few others were policy analysts for various social or political 
action and lobbying groups (e.g., the American Association for Retired Persons (AARP)). None of these 
persons reported having conducted inferential analyses of the NALS data, and so were not included in the 
second part of our survey. All of our survey responses were from individuals who are social scientists of 
one sort or another (e.g., educational researchers, economists, methodologists), and we therefore limited our 
analysis to those researchers who have conducted secondary inferential analyses of the data. 

We will identify the issues and concerns identified by these informants regarding their analyses 
and interpretations of the EBQ data and summarize their comments as they pertain to each of the six content 
areas of the EBQ. Unless otherwise noted, these comments are based upon our survey informants’ 
responses, rather than our own perspectives and views. 

Section A: Language Background 

This first part of the EBQ consists of 17 items that obtain information on NALS respondents’ 
acquisition and usage of English. Only one comment was obtained regarding the characteristics of this 
portion of the EBQ. It was suggested that English language proficiency, rather than what people can do in 
other languages — such as Spanish — should remain the focus of the NAAL. . We are aware that there has 
been some attention devoted to a Spanish- language version of the NALS (Solorzano 1994). Reynaldo 
Macias, a member of the NALS literacy definition and technical review committees, was lead author on the 
language minority report, which is in draft final form with the American Institutes for Research (R. 

Macias — personal communication, February 1 1, 1999). Two new variables — bilingualism and biliteracy — 
were advocated in this report to draw attention to the fact that many individuals — particularly immigrants — 
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may be literate in more than a single language (English), or not literate in English but literate in their native 
language (e.g., Spanish) or other languages not assessed by the NALS. It appears likely, however, that 
assessment of English language skills will continue to be the focus of the NAAL. 

It is our observation that the format of the language background portion of the EBQ may be 
confusing for NALS respondents — particularly the contingency questions (i.e., items #A-6, #A-7, #A-8, #A- 
9, #A-10) and the content of the hand cards. We have found, in our analyses, that it would be helpful to 
have an item that simply asks, “what is your primary language?” We will have more to say on this, and 
other issues, in the Recommendations portion of this report. 

Section B: Educational Attainment and Experiences 

This part of the EBQ has 14 items that obtain information about NALS respondents’ educational 
histories. A few of the informants to our survey suggested that there is a need for additional questions 
pertaining to educational and demographic variables. We will summarize their comments in the remainder 
of this section. A few informants indicated that more questions about previous educational experiences 
would be helpful. The item that indicates “some college” for educational attainment (i.e., items B-l, options 
G, H, and I), for example, is too vague; many more people have “some college” experience or credit hours 
than have a degree, and more needs to be known about these individuals’ educational histories. Exactly how 
much is “some college” experience, in terms of credit hours earned or years of schooling completed? It 
seems important to be able to quantify the varying amounts of college experience that respondents indicate; 
knowing these amounts would allow for better comparisons to high school graduates who have some 
technical school experience or a certificate, college graduates, and others. 

While it was possible to find data to describe the prose, document, and quantitative (PDQ) literacy 
of community college students, it was not possible to provide sound reasons why these students’ literacy 
proficiencies were at given levels. Several persons noted that the data on prior educational experiences, for 
example, were limited. Many college students are assigned to remedial courses, which are presently of 
much interest to educators. (See Breneman and Harlow 1999.) More specific information about respondents 
who are or were in remedial education would be helpful, according to one informant. Focusing on “highest 
education achieved” does not allow one to study those who were enrolled in community colleges. Just as the 



8 




NALS considers literacy as a continuum, educational attainment should also be viewed along a continuum, 
thereby avoiding distinctions about education. When the focus is only on the highest degree or educational 
status, this disregards the multiple and interacting factors that contribute to educational achievement. One 
informant indicated that more detailed information about NALS respondents’ educational background and 
experience would be very useful — particularly at the state level. For example, if a high school diploma was 
awarded, was it earned in regular or special education? Given the large percentage of individuals scoring at 
the two lowest proficiency levels, it is reasonable to expect many of these persons to have been recipients of 
special education services. 

Other types of credentials earned outside of formal education might be useful to know as well. This 
suggestion was made by a few informants — in particular, those who examined relationships between 
education, work, income, and literacy. These credentials could include craft apprenticeships and the like, 
which include the acquisition of skills that may be valued in the marketplace, but do not result in a degree or 
diploma, and may or may not lead to any kind of certification. NALS respondents’ participation in various 
forms of adult education would be very useful information for researchers, given the meteoric rise in 
participation rates over the past decade. 

Another informant commented that there was a need for more data on learning-disabled (LD) 
adults within the population. According to this person, some data suggest that as much as 1 7 percent of the 
U.S. population may have one or more learning disabilities. Those individuals at NALS proficiency levels 1 
and 2, who have been the subject of most concern to politicians, policymakers, and adult educators, are the 
most likely to have a learning disability (and to have been in special education) (Vogel and Reder 1998b). 
Therefore, a revised EBQ should have an increased focus on assessing the presence of LD within the U.S. 
adult population, according to one informant. 

A question of particular interest concerns how these individuals’ learning disabilities were 
diagnosed. Diagnoses of LD may be done by psychological (e.g., school psychologist) or medical experts, 
and the type of diagnosis will largely determine the LD treatment or intervention. It would be useful to 
know when the LD diagnosis occurred (e.g., age and/or grade level), if the respondent received any special 
education services while in school, their knowledge of the Americans with Disabilities Act (ADA), and if 
they had requested any school or workplace accommodations based upon ADA guidelines. Other questions 
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of interest regarding the LD population include: the impact of LD on employment; how respondents 
compensate for their learning disabilities in regards to literacy and work; and how the learning disabilities 
impact their personal lives. 

Another informant reported that the educational background and experiences data were very 
helpful to them in understanding the results — particularly at the state level — with regard to the association 
of educational attainment with literacy outcomes. A few others, on the other hand, suggested the NALS 
state-level data are very limited, and considered it best to use the national data for analyses and reporting. 

We suggest that two questions regarding educational attainment would help to clarify matters 
pertaining to educational achievement. One question should ask for the number of years of school 
completed. The second question should ask for the highest degree earned. We will address, in a subsequent 
section of the paper, what we view as the limitations of the EBQ regarding the items pertaining to NALS 
respondents’ participation in basic skills education. Suggestions for increasing the validity of these items 
will be provided. 

Section C: Political and Social Participation 

This portion of the EBQ consists of only four items that pertain to sources for obtaining 
information, television viewing, use of libraries, and voting. One informant referred to the lack of a question 
related to voter registration. The 1985 Young Adult Literacy Survey (YALS) (Kirsch and Jungeblut 1986) 
contained questions regarding both registration and voting, which allowed for modeling of voting behavior 
after controlling for the non-random selection of respondents (i.e., respondents can’t indicate “yes” to 
voting without being registered to vote). Those who register are not a random sample of the population of 
those persons who are eligible to vote on basis of age and citizenship. Also, items pertaining to NALS 
respondents’ involvement in different volunteer and civic activities (aside from voting) would be 
informative, and might provide additional insights about literacy practices. 

Combining the few items from this section of the EBQ with other sections may be worthwhile. For 
example, two items (#C-1, “how much information do you get from...” and #C-3, “how often do you use 
the services of a library...”) are directly relevant to literacy practices and could be included in the section of 
the EBQ that assesses such activities. 
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Section D: Labor Force Participation 



The fourth section of the EBQ consisted of 12 items that gathered information about the type and 
extent of NALS respondents’ occupational activities and income. An informant noted that there were no 
questions asked about the occupations of the parents of the NALS respondents. Because parents’ 
occupations will have some impact on the educational opportunities available to children, influence the 
kinds of literacy practices to which their children may be exposed, and provide models for participation in 
the world of work, it is important to determine parents’ primary occupations. Such an item could be 
included in the Demographic Information section of the EBQ. Also, it is critical to know respondents’ 
hourly wage rate. This information can be discerned from responses to items #D-3, #D*4, #D-7, and #D-8, 
but a single item which obtains this information would be useful, according to one informant. 

Another informant indicated that there is a need for good variables pertaining to income, but the 
relevant issues were somewhat different for this individual. According to this informant, it would helpful for 
individual states to know, from among those respondents who are receiving welfare benefits, the categories 
of welfare benefits received, e.g., food stamps, medical assistance, and/or cash assistance. Some interesting 
analyses might emerge from such data — at a basic level, it would be possible to determine the literacy 
proficiencies of individuals receiving various levels and types of welfare support. This information may 
have some important policy implications, particularly with respect to “welfare-to-work” initiatives. If, for 
example, those at the lowest PDQ levels are receiving the highest levels of welfare benefits, this may 
indicate that these persons are least likely to be able to enter the workforce without substantial basic 
education training. 

One informant, an economist, noted that “the data are extremely useful for economic analysis,” but 
would have found it more fruitful to have additional questions about the work history of the individual — the 
pattern of their previous employment and/or unemployment, how much they earned over each of the past 
five years, and what occupations they have held. A similar point was raised by a second economic analyst. 
Such information would allow for analyses of the impact of education on job mobility, according to both of 
these informants. 
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Section E: Literacy Activities and Collaboration with Others 

This portion of the EBQ contained only six items regarding NALS respondents’ uses of different 
kinds of text sources, such as newspapers and documents. A number of comments pertaining to this section 
of the EBQ were garnered and it appears that the information resulting from these items has been widely 
used. We have, of course, used the literacy practices data extensively in our studies (Sheehan-Holt and 
Smith in press; Smith and Sheehan 1998; Smith 1996). We therefore have several issues and concerns about 
the literacy practices items on the EBQ. While we will describe the issues more fully in a subsequent 
section of this report, we will briefly mention a few concerns here. First, the EBQ format pertaining to 
newspaper reading did not allow for a straightforward quantification of the amount of reading. Second, the 
item pertaining to book reading did not allow for estimates of the number of books read within given 
categories. 

Technology use has dramatically increased in both the workplace and in the home since the NALS 
was carried out in 1992. For example, the advent of the World Wide Web on the Internet, and its 
consequent availability to the mass public, has resulted in millions of Americans going “online” for 
information, education, commerce and entertainment. How, then, does computer and other technology use 
impact adult literacy skills and practices? What technologies are used and how frequently? What impact 
does technology use have for adults with learning disabilities? How do other media, e.g., listening to books 
on tape or viewing instructional videotapes to improve language or literacy skills, impact literacy? Thus, 
there is an expressed need for items that pertain to uses of electronic communications, computers, and other 
technologies for literacy uses and practices. 

An informant very familiar with the NALS development and the resulting data argued that there is 
a need for much greater detail regarding literacy practices. By asking respondents to provide specific 
examples of the typical, routine or daily literacy practices in which they engage, for example, this kind of 
information can help to validate their self-reports on literacy activities. It is critical for educators, 
researchers and policymakers to know what kinds of text materials adults use and the kinds of literacy tasks 
in which they typically engage. 



12 



20 



Perhaps even more relevant, according to this informant, is to determine the literacy tasks that 
adults face in specific settings and situations. For example, it is likely that, in most literacy situations, 
people must utilize two, three or more kinds of materials. They not only have to use these materials to 
accomplish the task(s) at hand, but they have to decide how to use this text or that one and often must gather 
information from multiple — and sometimes conflicting — sources. The EBQ does not capture this kind of 
real-life complexity of literacy tasks. This informant suggested that efforts be made to determine the nature 
and types of challenging literacy tasks that people face by surveying about 10 percent of the NALS sample. 
A diary for recording daily reading and writing activities might work well for this purpose (Anderson, 
Wilson, and Fielding 1 988). This approach would result in a better understanding of how well people can 
accomplish more difficult literacy tasks and activities. 

Another informant suggested that a revised EBQ should not contain the detailed question (item #E- 
2) pertaining to which sections of the newspaper are read (e.g., main news, financial and editorial pages, 
sports). However, because newspaper reading is the most pervasive and common reading activity among 
adults (nearly 9 out of 10 adults read a paper on a regular basis; Smith 1996), it is important to know as 
much about this particular reading practice as possible. Newspaper reading may be among the most 
fundamental behaviors for individuals to obtain information about a wide variety of topics and issues (e.g., 
politics, sports, finance, community development, parenting, entertainment activities) in a relatively simple, 
easy and accessible manner. 

Others suggest that, rather than eliminating items from the EBQ, it is very important to retain the 
literacy practices items. These items are important for determining what individuals with low education 
attainment and low literacy proficiencies are doing with the literacy skills they possess. Another informant, 
who has worked extensively on the International Adult Literacy Survey (IALS) described several 
dimensions of literacy practices that pertain to (1) the incidence and frequency of these activities; (2) the 
criticality of the practices; and (3) their complexity or challenge to the reader. At best, the NALS 
background questionnaire only captures the incidence and frequency of practices. Research by the first 
author (Smith in press) examined the reading practices among a sample of 150 adults and illustrated that it 
is essential to obtain the kinds of information called for above in order to more fully understand adults' 
motivations for reading, how they respond to demands for literacy, and the nature of these demands (see 
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also, Kirsch and Guthrie 1984). It might be possible, for example, to obtain more extensive reading 
practices data among a sub-sample of NALS respondents in future surveys, as suggested above. We will 
return to this issue when we discuss our recommendations for the next generation EBQ. 

Two informants suggested the need for a cleaner distinction between job literacy and social 
literacy practices (i.e., personal reading practices outside of the workplace), as these are likely to be 
motivated by different demands, require different text sources and result in very different outcomes and 
satisfactions for individuals. A more extensive assessment of literacy activities on the job, in the home, and 
in adult education settings would be very useful. 

Section F: Demographic Information 

The final part of the EBQ contained 1 3 items that gathered information on NALS respondents* 
marital status, gender, family and personal income, racial/ethnic identity and age. We obtained few 
comments from our survey respondents in regards to this section of the EBQ. Generally, these data were 
used for descriptive purposes in the various secondary analyses conducted by researchers (e.g., age and 
racial group differences in reading practices), and were widely used by nearly all of those who conducted 
secondary analyses of the NALS data. 

General Comments 

Several comments and suggestions were of a more general nature and did not pertain to specific 
content categories within the EBQ. We will summarize those comments here. 

One informant remarked that, in those cases where specific background items had very few 
respondents within a category, then the sample is obviously too small to make any meaningful inferences. 
Further, while in some cases, smaller groups can be combined for data analyses to allow inferences, this 
approach lacks the level of precision needed for adequate data interpretation. Two informants noted that, 
overall, the design of the EBQ is weak (but offered no information about specific weaknesses of the design) 
and one of them suggested that a short form of the literacy assessments (either combining PDQ into a single 
scale or simply using one scale such as Prose) could be used for the NAAL 2002. Obviously, Reder’s 
(1998b) analyses, which indicated that the three literacy scales are highly correlated with one another (rs > 
.93), is relevant here. A more extensive EBQ could be developed and used to obtain more detailed 
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background information. However, another informant indicated that it is unnecessary to add items to the 
EBQ. If anything, make it shorter and easier to complete in order to save time and money. A larger sample 
could then be obtained. Finally, one informant suggested that a much more extensive background 
questionnaire based on the Young Adult Literacy Survey (Kirsch and Jungeblut 1986) is needed for the 
NAAL. 

Summary 

We obtained complete surveys via e-mail, on the telephone and in person from eleven individuals 
who have employed the literacy proficiencies (PDQ) data from the National Adult Literacy Survey (NALS) 
and various data from the NALS English Background Survey in their investigations of adult literacy. The 
purpose was to ascertain the following: how these social scientists have used the EBQ data in their analyses; 
how the relative strengths and weaknesses of the data have enhanced or hindered their research questions 
and policy recommendations; and, how their suggestions for changing the EBQ to create an instrument (and 
resulting data) would be more useful to the NAAL 2002. 

It is clear that, although the numbers of individuals who have utilized the NALS EBQ data for 
inferential analyses is very small (in contrast to other national databases, such as the NELS:88 which has 
resulted in hundreds of studies over the years), the NALS data have been extensively used by these 
researchers. It is not clear, from our analysis, how extensively the data have been used by those who are not 
social science researchers but who nonetheless have a significant stake in adult literacy — local and state 
directors of adult literacy programs, for example, or community leaders and politicians, policy analysts, and 
leaders of business and industry. Anecdotal evidence, however (Smith and Reder 1998), suggests that uses 
of the data by these persons is restricted to descriptive summaries and does not involve detailed, inferential 
analyses of any EBQ data. It would be interesting, and useful, to learn more about how persons in these 
positions have utilized the NALS data to suit their needs and the needs of their constituents. 

The informants to our survey indicated use of the NALS EBQ to address a wide variety of 
questions and concerns. They also indicated that, while the EBQ has several virtues, there are nonetheless 
significant problems and limitations of the data derived from the EBQ. A number of suggestions regarding 
changes to the next generation EBQ were offered with the goal of gathering more useful and informative 
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data that enable stronger inferences. These suggestions pertain to all sections of the EBQ and primarily 
called for increasing the number of items to obtain additional and more detailed information. 

In the next section of this report, we describe our review of the relevant NALS studies that have 
utilized the EBQ data. For the most part, these studies were peer-reviewed prior to publication in a variety 
of professional journals. A few reports are book chapters; one is a technical report submitted to the National 
Center on Adult Literacy. 

Content Review 

As indicated above, several studies that involve secondary analyses of the NALS data have 
appeared in the social sciences literature. We conducted a content review of the empirical reports that 
utilized specific EBQ variables in the analyses. Our approach was to evaluate the researchers’ abilities to 
make strong inferences from these data, based on the results of their analyses and interpretations of 
findings. 

We examined the research questions for each paper, the methodology employed, the relevance of 
the NALS data to the research questions ( based either entirely or in part on NALS), the specific sections of 
the EBQ and specific items used for the analyses, and the results coupled with researchers’ interpretations 
of their findings. 

Criteria 

The following general criteria were used to review these studies: 

1. Is the research method appropriate to address the research questions? 

2. To what extent do the EBQ data allow for strong inferences on the investigators’ part? 

3. How did the strengths or weaknesses of the EBQ data affect the researchers’ policy 
recommendations (if any)? 

Review of Studies 

Fourteen studies, in which various portions of the EBQ data were employed in the researchers’ 
analyses of adult literacy, are reviewed. The papers are, for the most part, described in alphabetical order 
(by author). Friedman and Davenport (1998) examined gender differences in NALS literacy proficiencies, 
by age, educational attainment, and racial/ethnic groups. The research objective posed by the authors was a 
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general one — to explain patterns of gender differences in literacy performance, taking into account age and 
ethnicity. The NALS was the only data source for this investigation, and the EBQ data were important to 
the study. T-tests were used to compare males and females overall, and then separately for different age and 
ethnic groups represented in the sample. The relevant EBQ section pertained to respondents’ demographic 
information (age and race/ethnicity). 

There were significant gender differences among the 40-54 year old adults and those aged 65 to 80 
on Quantitative literacy only. There were a few gender differences on Quantitative literacy for whites only 
(for ages 40-54 and 65-80, favoring males). The researchers were able to make reasonably strong inferences 
from the data by examining differences among the age and racial/ethnic groups. They did not, however, take 
advantage of the large array of control variables in the NALS. In fact, they stated that the purpose of their 
study was to focus only on the variables that have been previously examined in the research literature. Their 
data analytic approach was exploratory in nature (i.e., they used the first plausible value for each PDQ 
variable, rather than the plausible value methodology proposed by Mislevy, Johnson, and Muraki 1992). 
Descriptive statistics were used to discuss differences among age cohorts. 

Friedman and Davenport concluded that educational attainment accounted for most racial/ethnic 
differences in literacy proficiencies, and that, as a nation, we are “doing better” at educating racial and 
ethnic minorities. In regards to gender, young women’s performance on the NALS illustrates the strides that 
females have made in academic achievement in recent decades. They also claim that the NALS results 
illustrate that cultural, rather than genetic, factors account for gender differences. No policy 
recommendations were made regarding methods to further close the “gender gap” in literacy. 

Gerber and Finn (1998) examined the relationship of literacy practices at both school and work 
with document proficiency. They note the discontinuity between work and school literacy and ask: (1) to 
what extent is document proficiency related to schooling? and, (2) to what extent are document skills 
acquired in the workplace? This study was based entirely upon the NALS data and the EBQ data were 
highly relevant. Information from the following EBQ sections was necessary to conduct their analyses: (1) 
educational background; (2) labor force participation; (3) literacy activities; (4) language background; and 
(5) demographic information. Tests of mean differences and multiple regression analyses were performed, 
and descriptive statistics were used to address the research questions. Gerber and Finn found, first, that 
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schooling is associated with better performance on document-related tasks, as adults with more education 
outperformed those having less education. Second, those who learned to read documents at work had at 
least as high a proficiency in document literacy as those who learned at school and in the community. Third, 
they found that reading documents at work is significantly associated with document literacy performance. 
Those adults who engaged in high levels of practice with work documents had the highest document literacy 
scores. Also, type of occupation is significantly related to document literacy skills. Higher-level occupations 
(e.g., professionals, managerial, technical, sales, and clerical) had higher document scores than lower-level 
occupations (e.g., service workers, farming, laborers). 

Their findings illustrate that the workplace provides a context for adults to acquire document 
literacy. Thus, both school and the workplace are important contexts for literacy development. The 
workplace may, in fact, be a suitable alternative learning environment for many individuals (e.g., high 
school dropouts), although further research is needed to examine this hypothesis. Gerber and Finn were able 
to answer their research questions and their use of the EBQ data was very appropriate. They were able to 
make strong inferences regarding the relationship between workplace reading practices and document 
literacy proficiency. These findings led them to suggest that employers may need to be more proactive in 
creating on-the-job educational opportunities — for their workers — which has several implications for 
workforce policy development and workplace literacy programs. 

Greenberg, Swaim, and Teixeira (1995) analyzed workforce literacy among rural adults in contrast 
to urban adults, and also examined differences by age, educational attainment, race/ethnicity, and region of 
the country. Comparisons on PDQ scores were made by metropolitan areas (e.g., central city, suburban, 
medium, small) and non-metro, rural areas (classified as either urban-adjacent, urban-nonadjacent, or totally 
rural). Greenberg et al. were able to distinguish levels of urbanization because they could identify the 
county of residence for each NALS respondent. Literacy levels of rural (nonmetro) adults vary widely, but 
are quite low, on average (specifically, near the upper end of Level 2). 

Greenberg et al. analyzed the literacy skills of rural and urban adults in order to determine the 
extent to which rural workers are able to compete economically with urban workers. Because there is a 
difference between urban (metro) and rural (non-metro) workers on literacy, this gap suggests that most 
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rural areas have a workforce literacy problem — only 19 percent of rural adults were found to score at levels 
4 and 5 on prose literacy. 

Greenberg et al. regressed individual literacy scores on either a dummy variable for metro county 
or five dummy variables for the most highly urban county types along an urban-rural continuum. The 
resulting coefficients measured the extent to which mean literacy is higher for more urban counties than in 
the most rural counties (i.e., what they referred to as “the rural literacy gap”)- Next, they re-estimated these 
models using 33 independent variables measuring factors such as age, gender, marital status, education, 
parents’ education, race, ethnicity, native-born speaker of English, and region of country. Thus, data from 
the demographic section of the EBQ was particularly vital for these analyses. This modeling approach 
allowed the researchers to examine the impact of each of the independent variables on literacy, while 
holding other characteristics and urbanization constant. The authors also tested the interaction of the 
rural/urban variables with each of the 33 control variables. The resulting model coefficient provided an 
estimate of how each characteristic either widened or narrowed the rural gap in average literacy. 

Greenberg et al. used similar regression techniques to investigate the effects of literacy on 
employment status and earnings. They regressed individual employment status on literacy scores using 24 
control variables for labor market experience, gender, marital status, education, race/ethnicity, native-born 
speaker of English, region of country, and metro residence, among others. Greenberg et al. wanted to know 
what the most important demographic characteristics that depress literacy in rural areas are, and if these 
variables are amenable to policy interventions. Also, they wanted to determine whether there are any 
advantages to literacy for rural-dwelling adults. Higher age and lower education were found to contribute to 
lower literacy levels. The labor market rewards for literacy (e.g., income) are much lower in non-metro 
areas than in metro labor markets. Therefore, the demand for workers with good literacy skills is less in 
rural than it is in urban areas. Lower economic rewards for literacy in rural areas tend to depress literacy 
because individuals have less incentive to develop these skills; those with higher literacy skills go to urban 
area jobs, according to Greenberg et al. 

The researchers were able to make strong inferences from the data by employing a large number of 
background characteristics as control variables. They noted that it was very important that the county of 
residence was given, since this information was essential to determining the level of urbanization. The 
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researchers made some suggestions for rural education policy and closing of the wage gap for high-literate 
rural workers. The study illustrates the need to prepare the rural workforce to meet the increasing literacy 
demands of the workplace in both urban and rural settings. From a methodological standpoint, using a great 
variety and number of background variables derived from the EBQ was a particular virtue of the study, and 
strengthens the researchers’ inferences. 

Howard and Obetz (1996) examined the literacy proficiencies of community college graduates. As 
community college educators themselves, they were very interested in examining the relationship between 
community college graduates’ literacy skills and literacy practices as these pertained to (1) having 
graduated from a community college and (2) graduates’ acquisition of information from a variety of 
sources — television viewing, listening to the radio, reading newspapers and magazines, and from family and 
friends. Their analyses are straightforward and, for the most part, descriptive. They compared the 
percentages of high school, community college and baccalaureate graduates at each prose and document 
literacy level. In regards to literacy proficiencies, they found that community college graduates perform “in 
the middle,” that is, higher than high school graduates but lower than college graduates. The authors did not 
appear to use much of the demographic information to tease out the other factors in the participants’ 
backgrounds, besides educational attainment, which might have important effects on assessed differences in 
literacy. 

Community college students’ literacy practices were examined, but were not compared to other 
education levels. Howard and Obetz offered several suggestions for community college educators regarding 
the improvement of their students’ literacy skills and practices (e.g., “use... students’ background 
information... to develop comprehension and writing skills,” and “developing. ..students’ awareness of 
interest in reading for self-improvement or recreation,” 467). The primary EBQ sections relevant to this 
study were for demographic information and literacy practices. Because the EBQ item that assesses the 
amount of personal use writing one does (#E-7) is largely limited to the types of writing one does at work 
(e.g., letters, memos, forms and reports), inferences from these data are not as strong as they would be if the 
item measured the types of personal writing individuals’ engage in at home and in other nonwork settings 
(e.g., writing recipes, taking phone messages, making grocery lists, journal writing). Further, since this 
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study is largely descriptive, the researchers should be cautious in making inferences that go beyond the 
NALS sample. 

Pryor and Schaffer (1997) examined the question of why university-educated adults have been 
found, increasingly, to take jobs requiring high school level skills. This trend has resulted in rising wages 
for these jobs. Their research question is as follows: Who are the university graduates that are ending up in 
high school jobs? Although it is somewhat unclear from their description, their analyses appeared to be 
based largely on data from the Current Population Survey (CPS) for 1971, 1979, 1987, and 1995. NALS 
EBQ data may also have been used to examine the numbers of university -educated adults in high school 
jobs. They then used the NALS PDQ scores to link ‘'functional literacy” to labor force status, occupation, 
and wages. Literacy proficiency scores and hourly earnings were compared across four levels of educational 
attainment (high school dropout, high school diploma only, some university courses, and university degree). 

Pryor and Schaffer show that it is primarily those university graduates who lack university- level 
literacy skills who are taking high school level jobs. Also, it is mostly those who are university-educated 
and in jobs requiring university-level literacy skills who are obtaining the major wage increases. This 
analysis resolves the “paradox” of why the apparent surplus of university graduates is associated with the 
rising wages of this group. 

For the purpose of linking functional literacy to labor force status, occupation, and wages, NALS 
proficiency scores were used by taking the average of the three PDQ scales. First, they examined functional 
literacy data by education and occupation; they then looked at the relevant wage data. The data show that 
the functional literacy of workers with a given level of education increases as the occupational tier 
increases. Thus, those with a university education working in occupations in which most workers had less 
than a university education showed lower functional literacy than university-educated individuals in 
university-level jobs. This suggests that university-educated persons in high school level jobs took these 
jobs because they had lower literacy qualifications than other university-educated workers and could not 
obtain jobs in occupations commensurate with their education. 

Pryor and Schaffer note, and this is also confirmed by Gerber and Finn (1998), that it is possible 
that functional literacy can be influenced by on-the-job learning, thus the job itself can play a role in the 
relationship between occupation, education, and functional literacy. They concluded that, from 1971-87, 
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university-educated workers of all ages were more frequently taking jobs in which the average educational 
level required was much lower (i.e., high school education). Also, university-educated workers who have 
experienced downward occupational mobility have, on average, much lower functional literacy than do 
other university graduates. Finally, the increase in wages of university-educated adults who are pursuing 
higher-level occupations reflects a shortage of university-educated workers with the functional literacy skills 
that are commensurate with a university education. 

We were unable to confirm from our reading of the report, and Pryor’s responses to our e-mail 
survey, whether the background and demographic information used in the Pryor and Schaffer analyses were 
derived from the NALS EBQ or.the CPS data. The NALS proficiency scores were more crucial to their 
arguments than were the background data, in any case. Generally, descriptive analyses were conducted. 

Raudenbush and Kasim (1998) examined racial, ethnic and gender gaps in earnings and 
employment. Although educational attainment is now similar for American men and women, a large gender 
gap remains in earnings. Also, while African-Americans and Hispanic Americans have fewer years of 
schooling and fewer degrees than European Americans, large ethnic gaps also exist in access to employment 
and earnings, even after educational attainment differences are taken into account. Several explanations for 
these gender and ethnic gaps in employment and earnings have been offered, according to Raudenbush and 
Kasim. First, quantitative measures of education (e.g., years of schooling and degrees obtained) 
inadequately capture the cognitive knowledge and skills that are important in the labor market. Second, 
people vary in their preferences for different kinds of work, and therefore select occupations that have 
varied job security and earnings. For example, women tend to select jobs that pay less than jobs that men 
select. Raudenbush and Kasim tested these explanations by examining between- and within-occupation 
inequality in employment and earnings by gender and ethnicity. 

They addressed four questions in their investigation: (1) How important is cognitive skill, as 
indicated by the NALS literacy assessment, in reducing the risk of unemployment and predicting earnings? 
(2) How large are differences in cognitive skills between groups? (3) How important is group inequality in 
cognitive skills for understanding social inequality in labor-force outcomes? (4) To the extent that cognitive 
skill differences do not account for ethnic and gender differences in economic outcomes, does the remaining 
inequality lie between or within occupations, or both? 
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The key variables for their analyses were the following: (1) social origin (age, parent education, 



gender); (2) human capital (educational attainment, credentials obtained, labor force experience, and 
literacy proficiency); (3) occupation (based on Department of Labor classifications); and, (4) labor force 
outcomes (natural log of wages per week and employment status). Therefore, data from several sections of 
the EBQ were critical to their analyses. The data analyses included the use of both the ordinary least squares 
(OLS) method and two-level hierarchical linear modeling (HLM) procedures (i.e., both between and within 
occupations). 

Raudenbush and Kasim found that the independent contribution of literacy (cognitive skill) to 
earnings is important, but not very large. On the other hand, the independent contribution of education, after 
controlling for literacy, is only marginally significant. Also, they determined that most of the effect of 
obtaining educational degrees is independent of literacy (cognitive skills). Further, literacy was found to 
contribute to reducing unemployment, but the effect was not large. 

Gender gaps in literacy are small but vary slightly across ethnic groups (as noted by Friedman and 
Davenport 1998). However, the ethnic gaps in literacy are quite large between African-American, Hispanic, 
and Asian males as compared to European American males. Ethnic gaps in literacy among females is much 
smaller. The OLS model accounted for 57 percent of the variance in literacy, but the HLM analysis 
accounted for 95 percent of the variance in literacy between occupations and 41 percent of the variance 
within occupations. Thus, literacy appears to be a potentially important variable in understanding gender 
gaps in employment and earnings. 

The ethnic gap in earnings between African-American and Hispanic males as compared to 
European males is quite large, but is substantially reduced by controlling for adult literacy. However, for 
females there is no ethnic gap in earnings for these groups. Raudenbush and Kasim found that 40 percent of 
the expected difference in log wages between African-American.and white males, having the same 
education, labor force experience, and parent education, is attributable to the fact that African-American_are 
less likely than whites to work in favorable occupations. Literacy explains more than half of the between- 
occupation gap, but less than half of the within-occupation gap. Having lower literacy skills, therefore, 
denies African-Americans access to favorable occupations and helps to explain wage inequality on the basis 
of ethnicity. Adult literacy is an important explanatory variable in regards to SES and ethnic inequalities in 
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earnings, and is an explanatory variable in regards to ethnic inequality in employment and risk of 
unemployment. 

Gender gaps in earnings ranged from large (e.g., European Americans) to small (e.g., Hispanics). 
Controlling for literacy has little or no effect on this gender gap. Further, the majority of the gender gap is 
within-occupation, rather than between-occupation. Therefore, neither theories of gender-based 
occupational preferences or of discriminatory occupational segregation explain this difference. Controlling 
for literacy essentially eliminated the between-occupation component of the ethnic gaps in risk for 
unemployment. Ethnic inequality in employment opportunities is a within-occupations phenomenon, 
according to Raudenbush and Kasim. However, about two-thirds of the African-American versus European 
American male gap in earnings, and essentially all of the gap in unemployment, lies within occupations. 

Among Raudenbush and Kasim’s conclusions were that there are important differences in 
cognitive (literacy) skills among persons sharing the same educational backgrounds, and these differences 
are linked to prospects for employment and earnings. The quality of schooling and of nonschool 
environments must, therefore, be improved for Hispanics and blacks. (It is interesting to note that Reder, 

1 998b, found that inequities in literacy proficiencies may impede any achieved equities in schooling — e.g., 
educational attainment — between minority and nonminority groups. Thus, attention must be devoted to 
increasing literacy among minority group members.) However, the situation is different for the gender gap 
in earnings, since higher literacy is not a predictor of this earning discrepancy. This is consistent with the 
possibility of labor-market gender discrimination. 

Raudenbush and Kasim employed very elegant analyses to address each of their research 
questions. They were able to make strong inferences from the data because they (a) used many control 
variables, and (b) innovatively conducted the analyses to separate within- and between-occupation 
differences. They noted in their discussion, however, that although they determined that much of the 
earnings gaps are due to within-occupation differences, they could not discern if this result was due to 
various noncognitive factors. We assume that this refers to motivational or behavioral differences between 
the genders and among the racial/ethnic groups, which may account for some of the earnings gap. This 
suggests that assessing some aspects of motivation (as it pertains to literacy) on the next generation EBQ 
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might be worthwhile. Among the limitations of their study, Raudenbush and Kasim suggest that a better 
measure of cognitive skills would have perhaps enabled them to account for the wage -earning gap. 

Steve Reder has been one of the leading analysts of the NALS data, having published several 
studies regarding various aspects of the data (Reder 1995, 1998a, 1998b; Vogel and Reder 1998a, 1998b). 
For example, he and Susan Vogel have examined the relationships of educational attainment and literacy 
proficiencies among adults having one or more learning disabilities, as self-reported in the EBQ. 

The primary purpose of the research reported in Reder (1995) was to disentangle the interactive 
effects of learning disabilities, literacy, and education on a number of adult outcomes. Several descriptive 
results were also reported. Reder examined both the prevalence of learning disabilities (LD) in the adult 
population and its distribution among various age, gender, and ethnic/racial groups. Also, the educational, 
social, and economic correlates of LDs were examined. Quantitative models of the joint impact of LDs, 
literacy, and education on these social and economic outcomes were tested. 

Several regression methods were used to determine how LDs, education, and literacy mutually 
affect one another. A reciprocal effects model (e.g., educational attainment affects literacy; literacy 
acquisition affects subsequent educational attainment) was also used to assess the direct and indirect effects 
of LDs on a number of social and economic outcomes. Structural equation models (SEM) were used to 
distinguish the direct effects of LDs on literacy from the indirect effects of other variables on literacy. 
Essentially, Reder wanted to know whether or not learning disabilities affect a variety of economic and 
social outcomes beyond what would be expected from the lower levels of educational attainment and 
literacy proficiencies that are associated with LDs. The reciprocal effects model (e.g., literacy and education 
are assumed to mutually influence one another) fit the NALS data extremely well. Thus, Reder was able to 
distinguish direct and indirect effects of LDs on literacy, education, and several social and economic 
outcomes. 

Reder (1998b) also used SEM to test the effects of literacy selection and literacy development on 
adults’ literacy skills. As noted above, literacy selection is the influence of literacy skills on educational 
attainment. Literacy development, on the other hand, is the effect of educational attainment on literacy. A 
third, reciprocal effects, model was also tested. SEM was used because of the simultaneous effects of 
education and literacy upon one another. The variables used in the analysis were age, gender, race/ethnicity, 
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educational attainment, parents’ education, learning disability status, log of annual earnings, and PDQ 
proficiency scores. 

The reciprocal effects model proved to better fit the NALS data than either the literacy 
development or literacy selection models by themselves. The reciprocal effects model accounted for 40 
percent of the variance in educational attainment and 54 percent of the variance in literacy proficiency, 
according to Reder. Next, Reder used the reciprocal effects model to examine the effects of education and 
literacy on a single economic outcome (i.e., annual earnings). Again, the reciprocal effects model fit the 
NALS data well for predicting annual earnings. Overall, literacy development processes were found to be 
more robust than literacy selection processes. 

One of the limitations of the analyses, and corresponding results, is the use of the cross-sectional 
NALS data to model longitudinal outcomes (i.e., literacy development). Nevertheless, the reciprocal effects 
model fit the data well at a single point in time. The study demonstrated the need for longitudinal 
examinations of adult literacy acquisition. Clearly, Reder employed very appropriate data analysis 
techniques, taking into account the sampling design of the NALS by rescaling the sampling weights 
according to the estimated design effect and using plausible values methodology to estimate coefficients and 
standard errors (see Mislevy, Johnson, and Muraki 1992). Inferences were strong, mainly because of the use 
of a complex model that controlled for many variables that impact literacy and educational attainment. 
Policy recommendations were offered to the effect that educational equity must include discussions of both 
educational attainment and literacy proficiencies — particularly because minority wage differences disappear 
when literacy is taken into account. 

Reder, in collaboration with Susan Vogel (Vogel and Reder 1998a), examined the educational 
attainment of adults who reported having one or more self-reported learning disabilities (SRLD) in the 
NALS. Descriptive statistics were reported (e.g., SRLD prevalence rates, gender ratios) and several 
inferential tests were conducted using design-weighted data. This procedure compensates for the complex 
sampling design of the NALS. 

Among the findings was a significant two-way interaction of group (SRLD vs. nonSRLD) by 
gender on years of schooling completed. Having a SRLD had a large effect on educational attainment. The 
high school/GED graduation rate was only 48 percent within the SRLD group in comparison to 84 percent 
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in the nonSRLD group. Among the SRLD group, more women (55 percent) than men (42 percent) had 
completed at least a high school education. Vogel and Reder provide a number of conclusions to their 
findings and offer several implications for educational policy regarding the improvement of educational 
outcomes for students with learning disabilities. These include: (1) creating of meaningful school 
environments that ensure success for students with learning disabilities; (2) improving the transitions from 
junior to senior high and from secondary to postsecondary education, including helping learning-disabled 
students to become more aware of the many postsecondary educational opportunities available to them, and 
creating personal transition portfolios that help students retain documents pertaining to their learning 
disabilities; and, (3) identifying and offering follow-up services to learning-disabled students who drop out 
of high school. 

This study enabled the researchers to generate educational policy recommendations from the 
NALS data. The EBQ data were particularly useful in this regard, as essential data were derived from the 
EBQ, including educational attainment (Vogel and Reder could determine not only if, but when respondents 
dropped out of school), gender and self-reported learning disability. Their breakdown of the information 
from the educational background portion of the EBQ into various types of secondary and postsecondary 
educational experiences strengthened their inferences. However, the self-report nature of the learning 
disabilities identification item may have limited their inferences because self-identification can be 
unreliable. In contrast, in the National Longitudinal Transition Study (NLTS) cited by Vogel and Reder, 
learning disability was identified by the respondents’ school rather than self-reported. In comparing rates of 
learning disabilities between the NALS and NLTS, Vogel and Reder found large differences. 

In a second article, Vogel and Reder (1998b) conducted several inferential analyses to compare 
SRLD adults to adults without learning disabilities and, again, examined gender differences between both 
groups. Both likelihood-ratio t-tests and analyses of variance were used to compare variable means. The 
focus of analyses was specifically on observed differences in literacy proficiency. PDQ scores were 
averaged to obtain a single indicator of literacy. The results were parallel to those reported in Vogel and 
Reder (1998a), as there were significant differences in literacy between SRLD and nonSRLD adults. 
Although overall, men and women do not perform differently on literacy, nonSRLD men slightly 
outperform women, while SRLD women perform significantly better than SRLD men. This latter finding is 
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in stark contrast to other studies that have shown that school-age females fare much worse than males on 
reading achievement measures. Vogel and Reder attribute this discrepancy to gender bias in school 
procedures for special education referrals. 

Generally, their data analytic methods were appropriate and the EBQ data allowed for strong 
inferences regarding SRLD versus nonSLRD differences in literacy. However, since Vogel and Reder are 
making inferences regarding two intact groups, controlling for some additional background variables (e.g., 
family income, parents’ education) could have further strengthened their inferences. An obvious limitation 
of the data is the self-reported nature of LD identification. Vogel and Reder made few policy 
recommendations in this report. They note, however, that their finding that severe literacy problems persist 
well into adulthood for those with SRLDs has important implications for educators. Fortunately, research 
has demonstrated some effective methods for teaching reading to students with learning disabilities; 
classroom teachers should leam more about how to implement these methods. 

Venezky, Kaplan, and Yu (1998) developed a predictive model of adults’ voting behavior using 
the NALS data (see also Venezky and Kaplan 1998). Demographic characteristics (e.g., age, gender, 
race/ethnicity, educational attainment, household income), literacy proficiencies (e.g., prose literacy) and 
literacy practices (e.g., frequency of newspaper reading, frequency of job-related reading, number of 
magazines read, and amount of newspaper and book reading) were used to predict voting behavior using 
logistic multiple regression analysis. 

The researchers’ analyses were appropriate and included very sophisticated multivariate methods. 
The results indicated that the likelihood of voting increases with age, literacy proficiency level, educational 
attainment, and income. Females were slightly more likely to have voted than males, and whites were 
somewhat more likely than other racial/ethnic groups to have voted. However, when age, education, 
income, prose literacy proficiency, reading habits, and other variables were statistically controlled, blacks 
were found to have a significantly higher probability of voting — indicative of the importance of using 
multivariate procedures when analyzing these kinds of data. Education was the single most important 
predictor of voting behavior. Other differences in reading practices were reported. Generally, those who 
read more text materials (e.g., newspapers, books) were more likely to vote than those adults who read 
fewer materials. 
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The researchers’ inferences were limited due to the restrictions of the data. First, the cross- 



sectional nature of the NALS design does not allow for an examination of changes in voting behavior over 
time. Second, analyses of voting behavior were restricted because there was no question regarding voter 
registration status (e.g., “Are you registered to vote?”) as in the Young Adult Literacy Survey (YALS) 
(Kirsch and Jungeblut 1986). Also, the voting question was different from that posed in the YALS (i.e., 
voting in local elections was not included in the NALS — only in national or state elections). Therefore, it 
was impossible to compare (1) probability of voting in local elections and (2) results from the two surveys 
for the 21-25 year old cohorts (the age range of the YALS). 

Due to the wording of the question, inferences regarding educational attainment were also limited. 
The NALS EBQ asked for the respondent’s highest level of educational attainment. If, for example, an 
individual were to drop out of school in 10 th grade, but then earn the GED, that would be coded as 
equivalent to an individual who dropped out at grade 12 and subsequently earned the GED. The authors 
suggest that these kinds of data limitations restricted their inferences. Yet, they nonetheless put forth some 
policy recommendations, and they note that, in future studies, it would be interesting to investigate the 
impact of adults’ use of online materials on reading practices. We next turn to a description of three NALS 
studies that the authors of this report have conducted. 

Smith and Sheehan-Holt Studies 

We have conducted several analyses of the NALS using data drawn from the EBQ. These analyses 
have largely focused on respondents’ reading practices and literacy proficiencies. Data from Section B 
(Educational Background and Experiences), Section E (Literacy Activities and Collaboration) and Section 
F (Demographic Information) of the EBQ have been the primary data sources in these analyses, along with 
PDQ scores. 

Smith (1996) conducted a study examining associations between adults’ PDQ literacy proficiencies 
and their reading practices involving the uses of five types of print contents — books, newspapers, 
magazines, and personal and work-related documents. Comparisons were made between high- and low- 
activity readers within each text category in regards to literacy proficiencies. The cut points for 
distinguishing between the high- and low-activity readers corresponded approximately to that point dividing 
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the distribution in half, allowing for use of all of the available data. A secondary focus of the study was on 
age differences in reading practices, taking advantage of the cross-sectional design of the NALS. Both 
descriptive and inferential results were reported. The primary data analyses employed analysis of variance 
to test differences between high- and low-activity readers in regards to the different print contents. Multiple 
regression analyses were performed to examine age group differences while controlling for educational 
attainment. The findings can be summarized by saying that, in general, those who read more categories of 
printed materials — regardless of the nature of these materials — had significantly higher PDQ scores than 
those who read fewer categories of texts. Book readers demonstrated the highest PDQ scores. Both 
educational attainment and reading practices were found to contribute significantly to PDQ literacy 
proficiencies. 

Inferences about between-group differences could have been strengthened by taking into account 
important demographic characteristics (e.g., parents’ educational attainment, income). The study also would 
have benefited from having additional information pertaining to adults’ uses of reading materials and more 
detailed information about the materials themselves. For example, respondents were asked to indicate the 
number of categories of books read, rather than to estimate the total number of individual books read. 
Obviously, some individuals may read dozens of books within a single category, while others may read a 
few books from several categories. Thus, this estimate of book reading is a very crude measure of reading 
behavior. 42 percent of the adult population read from a book category at least once a week or more. Given 
the pervasiveness of book reading, it would be very informative to know more about the quantity and types 
of books read. Also, book reading was found to be higher for blacks than for whites when some background 
characteristics were controlled. It would, therefore, be interesting to know if this finding holds up for the 
total number of books read, or for number of books within certain categories. 

Although somewhat more difficult to obtain, data regarding the relative quality of the reading 
materials typically used by adults would also be informative. Some book and periodical readers read only 
highbrow literature and reputable news dailies and journals (e.g. the New York Times, Atlantic Monthly, 
Wall Street Journal); others’ reading tastes tend toward popular fiction (e.g., romance novels) and weekly 
tabloids; still others’ reading may lie somewhere between these two extremes. Asking respondents to 
indicate specific, prototypical examples of their reading on the EBQ will allow us to obtain highly useful 
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and interesting data and enable much more fine-grained analyses of adult reading practices and how these 
practices may contribute to literacy skills (this point was also raised by one of our survey informants — see 
above). 

The Smith and Sheehan (1998) chapter summarizes the Smith (1996) study and includes a second 
set of analyses by the two authors who examined racial/ethnic group differences in reading practices. No 
specific research questions were stated. A series of regression analyses were conducted. Age, education, 
occupation, and prose literacy, but not income, were controlled in all analyses. Several interesting 
racial/ethnic group differences were found with respect to reading practices that require further study. 
Recommendations pertained to increasing document literacy instruction in school, particularly for ethnic 
minorities, and noted that increased efforts should be directed toward developing the reading skills and 
practices of socially and economically disadvantaged adults in ways that lead to higher literacy skills. The 
authors could not make strong inferences regarding the literacy practices of Hispanics (those whose native 
tongue is Spanish) since the NALS assessed only English literacy skills and practices. 

Sheehan-Holt and Smith (in press) also investigated the association between adults* participation 
in basic skills education and their reading practices. The rationale for this study was to ascertain the role 
that basic skills programs may play in contributing to adults’ literacy. Obviously, in an ex post facto study 
of this type, there are no pretest measures to distinguish basic skills participants from nonparticipants. A 
number of control variables were therefore used in the regression analyses, including educational 
attainment, English as primary home language, presence of disability or long-term illness, and newspaper 
reading practice. 

The NALS EBQ contained three items pertaining to basic skills education. The first asked if the 
respondent had ever taken part in a program (other than in regular school) to improve their basic skills in 
reading, writing, and arithmetic. If the respondent indicated “yes,” s/he was then asked to identify the type 
of program in which s/he had participated. Four broad categories of programs were listed, including (1) a 
training program or courses given or sponsored by an employer or union, (2) a publicly sponsored 
education and training program such as Job Training and Partnership Act (JTPA) or Adult Basic Education 
(ABE), (3) a tutoring program sponsored by a library, church, or community organization, and (4) any other 
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program. Finally, they were asked how recently they had taken part in this training: still enrolled; within the 
past year; between one and five years ago; more than five years ago. 

No evidence was found that, as compared to nonparticipation, participation is related to better 
literacy skills. Basic skills participants did have higher reading practices in four of the five reading content 
areas. For example, adults who took part in job-related basic skills programs read newspapers and work 
documents more frequently than did those who participated in tutoring-type programs. The NALS limited 
the investigators’ ability to make inferences regarding possible outcomes when adults participate in basic 
skills programs, since there were no data regarding the duration of individuals’ participation in basic skills. 
This information would be very useful, as other studies suggest that upwards of hundreds of hours of 
instruction may be needed to positively affect literacy skills for those individuals at the lowest literacy 
levels (Sticht and Armstrong 1994). 

The researchers were able to address their research questions in each of the two studies above. 
Appropriate sampling weights and plausible value methodology (Mislevy, Johnson, and Muraki, 1992) were 
employed in every case and, in Sheehan-Holt and Smith, the design effect using HLM was taken into 
account. The nature of the EBQ affects the inferences made in all three studies. In the case of the Sheehan- 
Holt and Smith study of basic skills participation, much more information is needed in regards to the quality 
of basic skills programs and instruction, duration of participation in basic skills programs, and so on, to 
determine how such programs may impact literacy skills. The use of a large number of control variables 
helped to strengthen the inferences made regarding the differences between participants and 
nonparticipants, although this approach does not substitute for a longitudinal analysis. 

Summary of Content Review 

A total of 14 papers reporting analyses of the NALS data, including data that were drawn from the 
EBQ, were described in this section of our report. The foci of these studies pertained to the associations 
between literacy and education, gender, age, race, ethnicity, and voting behavior, inequalities in economic 
opportunities, employment and wages, and learning disabilities, among other topics. We identified the 
research questions, the nature of the data analysis procedures, and most significantly, the relative abilities of 
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the investigators to utilize data derived from the NALS EBQ to conduct their studies in a satisfactory 
manner. In other words, how adequately did the EBQ data contribute to these studies? 

The research questions are obviously constrained by the data available to the analysts. Thus, the 
researchers generally did not pose questions that they would be unable to answer with some degree of 
confidence. It is true, of course, that most would have preferred to have more variables to draw upon from 
the EBQ, and in some cases, entirely different variables to enhance their studies and shed further light on 
the explanatory variables regarding adult literacy. Nonetheless, these studies collectively go a long way 
toward furthering our understanding of the powerful role that literacy plays in American adults* lives in the 
late twentieth century. 

Conclusions 

What kinds of information should be obtained in a revised English-language Background 
Questionnaire for the NAAL 2002? The recommendations that follow are based on our survey of NALS 
EBQ data users, analyses of the published research reports (in the literature) that have used data from the 
EBQ, our own extensive experience with these data, and the recommendations of a series of focus groups 
conducted by the American Institutes for Research (AIR) in early 1998. We recommend several revisions 
and changes to the EBQ for the NAAL 2002. We discuss various technical and substantive implications of 
these recommended changes, our views on the impact of these changes in terms of costs and benefits to 
adult literacy stakeholders, and how changes to the EBQ might impact the assessment of trends in adult 
literacy. 

Technical and Substantive Implications 

How well do the background data enable researchers and policy analysts to make strong inferences 
regarding the effects of various factors on adult literacy? Our evaluation of the published studies that have 
utilized the NALS English-language Background Questionnaire data leads us to give the EBQ a “passing 
grade” in terms of the utility of the data for enabling strong inferences regarding literacy within the U.S. 
adult population. With the advent of sophisticated statistical software and methods in recent years — 
particularly hierarchical linear modeling (HLM) and structural equation modeling (SEM) — researchers are 
able to test various models to determine the nature of relationships among variables. The NALS data, 
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because of the sampling design employed and large numbers of variables and participants that characterize 
the survey, are well suited to these high-level statistical procedures and yield very robust results. 

Nonetheless, there are several important limitations and weaknesses of the EBQ data that have 
been acknowledged by the developers and pointed out by those who have subsequently utilized the data for 
research and policy. The identified problems resulted from the inevitable tradeoffs and compromises 
necessary to construct a measure which captures essential data within the practical and logistical constraints 
of conducting a large-scale and costly national survey. Clearly, the 1992 NALS should serve as a learning 
tool for refinement of the next-generation NAAL and every attempt should be made to correct the 
acknowledged flaws and overcome the perceived limitations of the EBQ. At the same time, changes to the 
current version of the EBQ must be weighed against the potential loss of opportunities to examine trends in 
adult literacy from 1992 to 2002. Such examinations will be significantly hindered if the EBQ is radically 
altered. Perhaps a reasonable compromise is to include additional items, as suggested by several of the 
researchers whom we interviewed, and to maintain most or all of the present items. This approach, of 
course, will impact the manner in which adults* literacy proficiencies are assessed. 

Recommended Changes to the EBQ 

The following comprises an extensive list of recommended changes — mostly additions — to the 
current form of the EBQ. Virtually every part of the EBQ would be altered by implementing these 
recommendations. The result would be more useful information for adult literacy stakeholders. We will 
describe the recommended changes section by section. 

General and language background . First, as previously discussed, a question should be added 
which asks the respondent, “What is your primary spoken language?” Phrasing the question in this manner 
allows the respondent to indicate his/her oral language, rather than having the data user try to ascertain the 
primary language. That is, currently, different data users can employ different language background items 
(e.g., items #A-4, #A-8, #A-15) to determine respondents* primary language — resulting in a lack of 
consistency in target populations across studies. Second, the AIR focus groups recommended that more 
information on language use and facility in other languages be obtained. This information was obtained in 
the 1992 survey, but was not disseminated. 
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Educational background and experiences. More information is needed about respondents’ 
educational histories and attainments. This seems particularly critical given the obvious relationship 



of formal schooling completed be added. The addition of this question captures all of the information 
presently obtained by item #B -6, except for the “vocational, trade, or business option.” A separate question 
could be posed to ascertain type of vocational, trade, or business degree, certification, or other credential 
earned, if any 

Because the NAAL 2002 will have substantial implications for educational policy as it pertains to 
adult education, it seems particularly important to gather more data on adults* participation in various basic 
skills training programs. This recommendation was also made by the AIR focus groups. Specifically, we 
recommend that, in addition to the current items, respondents’ estimates of the duration of their 
participation in basic skills programs (i.e., number of weeks enrolled) be obtained. Also, was the basic skills 
program completed? Because it is typical for people to enter, drop out, and re-enter these kinds of 
programs, it is important to ascertain the number of times that the respondent has enrolled in basic skills 
programs. For those adults who participated in one or more basic skills programs and completed the 
programs, did they subsequently continue their education by enrolling in college or a vocational, trade, or 
business school, or pursue any advanced form of occupational training? What other kinds of training and 
credentials that prepare individuals for the workplace were earned, aside from educational diplomas and 
certificates? 

Participants in basic skills education typically, but not exclusively, lack a high school diploma. 
People drop out of school at different ages and grades. An individual who drops out at 12 th grade, for 
example, may be better prepared to obtain the GED than a person who dropped out in 10 th grade. This is 
another reason for determining highest grade, or number of years of school, completed. In addition, to 
capture more complete information about this population, a question could be added that asked whether one 
obtained a GED and if so when. Finally, for those who have graduated from high school, how was their 
degree earned? In other words, did they receive any special services while in school (e.g., special education) 
or participate in talented and academic gifted programs? 



between education and literacy proficiencies. We recommend that an item that asks for the number of years 
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Political and social participation. It is our recommendation that this section of the EBQ either be 
(1) extensively expanded to capture more information about adults’ social participation as it relates to 
literacy, or (2) truncated by deleting the four sub-scale items along with other sections of the EBQ. In our 
estimation, only two items on this sub-scale pertain to “social participation” — TV viewing and library use. 
Thus, the scale extensively taps into neither adults’ involvement in society nor their communities where 
they might engage, more or less extensively, in literacy. Questions could be posed that ask more specifically 
about the extent of respondents’ community involvement and social networks. Examples include: 

“Have you ever: 

-written a letter to the editor of your local newspaper? 

-written a letter to, contacted by phone, or visited the office of your city mayor, state 
representative, Congressional representative, Senator, or other elected or appointed 
representative at any level of government? 

-volunteered to work in a civic, charitable, political or social organization? 

-been a member of a local PTA/PTO or other school organization? 

-been a member of a local church or parish?” 

Our view is that items #C-1 (“how do you usually get information. ..”), #C-2 (“how many hours do 
you usually watch television each day?”), and #C-3 (“how often do you use the services of a library. . .?”) 
are literacy practices questions that could be moved to Section E (Literacy Activities and Collaboration). 
Only item # C-4 (“have you voted in a national or state election...?) pertains to political participation. As 
noted previously, the item could be revised to include voting in local elections. Some people vote in local 
elections but not in state or national races; others vote only during national elections, but rarely for local 
ones. The primary problem with making this change is that it impedes trend analysis from 1992 to 2002 in 
regards to voting behavior, so these changes should be made only after careful consideration. We also 
suggest that a second item pertaining to respondents’ voter registration status should be added. Rather than 
having a section of the EBQ which contains only two voting status items, however, we recommend shifting 
these questions to the Demographic Information section. 
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The AIR focus groups recommended the inclusion of items regarding civic participation, such as 
how closely participants follow public affairs, and how they obtain this information. The assessment of 
political participation could also be enhanced by asking if respondents have ever (1) held public office, (2) 
volunteered to assist in a political campaign, or (3) been involved in any other forms of political activism. 

Labor force participation. We recommend several additions to this section of the EBQ in order to 
obtain data that will be useful to labor market economists, sociologists, and others. Based upon suggestions 
by researchers who have utilized the labor force data extensively, we recommend the addition of a question 
which asks respondents to provide their hourly wage rate (presently, this data can be determined by 
responses to items #D-3 and #D-7, although data from items #D-4 and #D-5 may also need to be used to 
determine hourly wage rate). Also, more questions about respondents’ work histories are important — what 
and how many jobs have been held over the past five years? Which jobs were full-time work? Which ones 
were part-time work? For part-time workers, is full-time work being sought? 

Additionally, researchers interested in labor force participation would have liked to have had more 
information on the work and wage histories of unemployed persons in the NALS sample. For unemployed 
respondents, it is critical to know something about their most recent job, when they were last employed, the 
type of occupation, and hourly wage rate. Several of these recommendations were made by the AIR focus 
groups. 

Literacy activities and collaboration. We recommend several additions and revisions to this 
section of the EBQ. First, given the dramatic increase in technology availability and use for the average 
citizen, questions should be included that gather information about the kinds of technology that respondents 
are using (e.g., computers), how frequently (e.g., daily, a few times a week), and for what purposes (e.g., 
work, personal, entertainment). The Internet, for example, has become a major information and 
entertainment medium in the years since the NALS data were collected and will become even more 
significant in everyday life for millions of Americans by 2002. How has Internet use impacted adult 
literacy? Who has access and how? These questions could be addressed by including the recommended 
items. 

The book reading practices item should be changed to capture not only categories of books read 
but also the total number of books — either a grand total or within each category. The magazine reading item 
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should be changed to reflect reading on a weekly basis, which is more specific than asking how many are 
read on a “regular” basis, which can have different meanings to people. Weekly reading is the same 
benchmark for all respondents. 

We recommend a cleaner distinction between reading for personal (or social) uses (item #E-5) and 
for work (item #E-6). The print content choices (e.g., letters or memos, reports, articles, magazines, or 
journals, etc.) are the same for both items. The kinds of reading done for work and for personal/social 
reasons appear to be very different in most instances, however (Guthrie, Seifert, and Kirsch 1986; Kirsch 
and Guthrie 1984). Allowing respondents to provide examples of their reading for both types of purposes 
(i.e., personal/social, work) would be very informative. This is particularly true for the items that pertain to 
writing activities (#E-7, #E-8). Only writing for personal use (e.g., writing letters, memos, forms, and 
reports) was assessed in the NALS. We recommend expanding this item (#E-7) to include examples of 
writing practices which are more typical of everyday life, such as making lists, taking phone messages, 
keeping a diary, and summarizing or getting the gist of a lengthy document or article. 

In fact, more information could — and needs to — be gathered about adults’ typical literacy 
practices, including reading, writing, and arithmetic. This means not only assessing the frequency and type 
of practices, but also learning about the challenge, or difficulty, of the literacy tasks that adults face, as well 
as the criticality, or significance, of these literacy tasks and activities to their lives and work. Currently, 
Statistics Canada is examining these issues, based upon what they have learned from the results of the 
International Adult Literacy Survey (IALS) (S. Murray — personal communication, March, 1999). These 
data could be gathered on a sub-sample using a diary approach. 

Demographic information. Information regarding the occupations of the respondents’ parents is 
needed. Parents’ occupational status clearly has a significant impact upon educational, social and job 
opportunities and, at least indirectly, has some influence on literacy. This kind of data would have been very 
useful to previous NALS-based research on the relationships among occupation, literacy, and wages. We 
also suggest that one or more items pertaining to voting and voter registration status be included in this 
section if the Political and Social Participation section is not expanded. 
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Alternative Approaches 



Increasing the number of items contained within the EBQ will necessarily increase the time needed 
to administer the survey. What can be done, then, to reduce the overall administration time for the literacy 
assessment and background survey? Some experts have suggested revising the literacy assessment. Reder’s 
(1998b) analyses have shown that the PDQ literacy proficiency scales are highly correlated with one 
another. In essence, each scale taps into somewhat overlapping dimensions of the same basic ability. These 
issues are, by now, well known to the NALS developers so we will not discuss them in detail here. It is 
sufficient to note that Reder’s evidence “indicates that the dominant feature of the assessed [italics in 
original] construct of literacy is a single general literacy capability” (52). Individual literacy items on the 
NALS scales were shown to cut across all three categories; thus, as Reder’s analyses suggest, it should be 
possible to assess and interpret literacy performance using a single scale. 

There has been some suggestion that, for example, only the document proficiency scale be 
employed in future NALS assessments. Individual items contained in the document scale may contain more 
or less prose and/or quantitative tasks embedded within them, and these items could then be subjected to 
further analyses to ascertain respondents’ prose and quantitative literacy performance, if so desired. Another 
approach is to combine all three assessments into a single general literacy scale. This may be a reasonable 
approach to reducing the administration time for the literacy assessment and thus allow for increased 
administration time of a lengthier EBQ. 

Sticht’s (1999) advocacy of a telephone-administered NALS-type literacy assessment also has 
implications here. The advantage of a telephone assessment, among other benefits that are described by 
Sticht, is the possibility of having multiple contacts with respondents. For example, a lengthier, more 
detailed version of the EBQ could potentially be administered in several stages to the same respondents. 
Multiple telephone contacts may have the effect of establishing greater rapport between the interviewer and 
the respondent and thereby yield a stronger commitment to participation on the respondent’s part and may 
result in more reliable data. Sticht has estimated an 80 percent cost savings for a national literacy 
assessment through the telephone survey method. Even if multiple telephone contacts with respondents were 
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required to administer a more comprehensive EBQ, and respondents were paid for each interview, the costs 
should still be well below that of the 1992 NALS. 

We are also informed by the current longitudinal study of adult literacy being conducted by Steve 
Reder. The longitudinal study is sponsored by the National Center for the Study of Adult Learning and 
Literacy, at Harvard University, and is funded by NCES. The purpose of the longitudinal study is to “better 
understand the process and consequences of adult literacy development over time” (S. Reder — personal 
communication, March 8, 1998). Information is being gathered about two groups of adults in the Portland 
(OR) greater metropolitan area who have low educational attainment — those who are enrolled in adult basic 
education programs (e.g., GED) and those who are not. Participants in both groups are going to be followed 
over three years and surveyed once each year. Participants in the former group are identified through the 
various programs in which they participate (n=500); those in the latter group are identified through a 
random-digit dialing telephone screening interview (n=500). In addition to a literacy skills assessment, a 
very extensive background survey (“lifelong learning questionnaire”) is individually administered using a 
computer-assisted personal interview script. Both the literacy assessment and background interview require 
about two hours to complete (S. Reder — personal communication, April, 1999). Participants are paid thirty 
dollars. There is some hope that this limited project will be the test ground for a national longitudinal study 
of adult literacy. 

The Lifelong Learning Questionnaire used by Reder’s research team obtains information on the 
following topics: 

• Educational background and school history; 

• Employment status and work history; 

• Contextual influences on learning; 

• Language background; 

• Learning motivations, contexts and activities; 

• Participation in adult education; 

• How information is obtained; 

• Reasons for dropping out of adult education programs; 

• Literacy practices at work, home, and school; 
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• Social networks and civic involvement; and, 

• Demographic variables. 

It is our recommendation that the NAAL 2002 development team look closely at the Lifelong Learning 
Questionnaire and consider incorporating a number of the items contained in this survey consistent with the 
above recommendations. 

The National Center for Education Statistics conducted, in 1991 and 1995, a National Household 
Education Survey (NHES), using random-digit dialing procedures. The 1995 survey consisted of two 
components: an adult education component and an early childhood program participation component. The 
adult education survey is relevant to our report, as the sampling and data collection methods are quite 
similar between the NALS and the NHES:95. Interviews were completed with nearly 20,000 adults ages 16 
and older who were not on active military duty for the NHES:95. This survey focused on the participation 
of adults in a wide range of educational activities during the year prior to the interview. Respondents were 
asked about their participation in seven types of adult education, including basic skills and GED classes, 
college coursework and degrees, vocational training, apprenticeships, work-related courses, and other forms 
of instruction. Also obtained were data regarding reasons for and barriers to participation in adult education. 
A few literacy-relevant variables were also obtained (e.g., public library use). 

Our past work with the NALS (see Smith and Sheehan 1996) indicates that variables obtained at 
the different levels of sampling might be important to understanding some of the relationships among 
literacy and other variables in the NALS. We have found that the intraclass correlations of the three literacy 
scales are moderate at the segment level. Because the segment level of the sampling design constitutes 
blocks or groups of census blocks, we have termed this segment-level variation “neighborhood effects” (see 
Sheehan-Holt and Smith 1998). Our work has determined that the mean income of the neighborhood 
accounts for variation in literacy skills among individuals — even beyond what is accounted for by family 
income. Therefore, another recommendation which developers of the NAAL 2002 should consider is the 
collection of data at higher levels of sampling design — particularly the segment level. For example, such 
data might include the crime rate of the neighborhood, the percent of African Americans and Hispanics in 
the neighborhood, whether there is either a library or other resources for literacy available in the 
neighborhood, and the average family income and average education level of the neighborhood. 
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It may be worthwhile to consider obtaining more extensive literacy-relevant background data (e.g., 



literacy activities and collaboration) in future NCES adult education surveys. Perhaps the NAAL 2002 
could be combined or integrated with a future adult education survey or vice versa — a change that was also 
called for by the AIR focus groups. A sub-sample of the NAAL 2002 participants could also complete the 
NHES adult education survey. Integrating the EBQ and the adult education surveys would result in a 
lengthier EBQ which could be administered over the telephone, across multiple contacts, as suggested 
above. Combining the two surveys achieves the purposes of both the NAAL and the NHES and can result in 
substantial cost savings. Linking information ffom both surveys would enable researchers to study how 
barriers to adult education may, for example, deter the acquisition or development of literacy among some 
populations. 

The AIR focus groups recommended several changes for the NAAL 2002, as we have previously 
noted. Among their recommendations is that a mechanism should be used to “relate the 2002 survey results 
to the demographic and dynamic changes in society” (Sherman, Condelli, and Koloski 1998, 6) and to 
report the information in a broader context. It may not, however, be possible to implement this valuable 
recommendation without the benefit of additional background data. We also note that the consensus of the 
focus groups was to expand the EBQ to obtain more descriptive information about the sample, thereby 
enabling more in-depth analyses by primary and secondary analysts. 

Assessment Costs 

Despite the recommendation — by the writers of this report and other adult literacy stakeholders 
and experts — for the addition of numerous items to the EBQ, we recognize the need to have a parsimonious 
set of background questionnaire items that will provide the most essential information about literacy. The 
data must be sufficiently robust and detailed to allow strong inferences about adult literacy. There is a 
delicate balance between managing the costs of a survey (which includes administrative and other 
personnel, and data management and analysis costs) and obtaining good data. The advantages of adding or 
deleting items and/or modifying existing items contained within the EBQ must be weighed against the 
disadvantages (i.e., the potential costs — in terms of real dollars and/or the sacrificing of useful data). 
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The addition of items to the EBQ will increase the administration time for the survey and the 
overall costs of data collection. Both interviewers and respondents must be paid for their time, among other 
costs. If the administration time is increased for the EBQ, then it may be necessary to decrease the time 
allocated for the literacy assessment to avoid imposing unreasonable demand upon respondents (which 
would make it more difficult to obtain participants — thereby further increasing data collection costs). 
Further discussion will be necessary to determine the best course of action in regards to the literacy 
assessment strategy. 

The primary advantage to changing the EBQ used in the 1992 NALS is that, potentially, more 
useful data can be obtained as a result of these revisions. Any content changes to the EBQ may present 
complex problems that impede trend analyses. Therefore, the recommendations that involve the addition of 
new items are not as dramatic as those that call for some items to be revised. However, the addition of items 
can have unintended effects. That is, the antecedent items may suggest a particular frame of reference for 
subsequent questions and, therefore, responses to these latter questions may be influenced because the 
context has been changed. These changes, if implemented, will present a particular challenge to the EBQ 
developers for the NAAL 2002. 

It is very important for researchers and policymakers to have an expanded EBQ from which to 
draw data. Several researchers have expressed to us that was (and remains) frustrating to have such a 
limited EBQ and — at the same time — a very good assessment of literacy proficiencies. Because a great 
amount of effort, time, and dollars have been committed to developing a quantitative measure of literacy, 
we believe that it will be beneficial to all literacy stakeholders to have an EBQ that will allow researchers to 
conduct more thorough investigations of the many factors that are important to adult literacy. 
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A. Initial Contact E-mail Survey 

Dear NALS Data User: 

Your name appears on a list of academic researchers, public policy experts, and other professionals who are 
users of the 1 992 National Adult Literacy Survey (NALS) data 

[http://nces.ed.gov/nadlits/naal92/users.html]. We are currently preparing a technical report for the 
Education Statistics Services Institute (ESSI), evaluating the uses of background questionnaires in survey 
studies such as the NALS, with recommendations for changes in the background questionnaire for the 
National Assessment of Adult Literacy planned for 2002. As part of this effort, we would like to ask for 
your assistance. 

The NALS background survey consisted of six “content areas” that gathered information on respondents’: 
(1) language background; (2) educational attainment and experiences; (3) political and social participation 
(e.g., voting); (4) labor force participation; (5) literacy activities and collaboration with others; and (6) 
demographic information. 

We would like to know three things: 

First, have you conducted any analyses of the NALS data, or in any way reported the data from each or all 
of the six content areas contained in the background survey? 

Second, if you answered “yes” to the previous question, which among the six content areas do you consider 
to be your area of expertise? (You may indicate one or more areas). For example, if you have analyzed 
relationships among educational attainment and literacy proficiencies, indicate “educational attainment” as 
your area of expertise. 

Third, would you be willing to respond to a brief, follow-up e-mail or telephone survey? Please indicate 
your preference for either e-mail or telephone follow-up. 

Your responses to this brief survey, and the follow-up survey, are and will remain anonymous. 

Thank you for your assistance. 
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M Cecil Smith, Ph.D. 

Associate Professor 
Department of Educational Psychology 
Northern Illinois University 
DeKalb, IL 60115 

(815) 753-8448 
mcsmith@niu.edu 



Janet K. Sheehan-Holt, Ph.D. 

Assistant Professor 

Department of Educational Psychology 
Northern Illinois University 
DeKalb, IL 601 15 

(815) 753-8523 
jsheehan@niu.edu 
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B. NALS Background Survey Data Users Questionnaire 

You may recall that we recently contacted you and asked you to respond to our brief “pre-survey” 
pertaining to your use of data from the National Adult Literacy Survey. At that time, we indicated that we 
wished to follow-up with a somewhat more extensive survey regarding your uses of the Background Survey 
from the NALS. We are preparing a technical report for the Education Statistics Services Institute in their 
preparation for the planned 2002 National Assessment of Adult Literacy. Results from the survey below 
will be incorporated into our report and used to make recommendations for the development of the National 
Assessment of Adult Literacy. We appreciate your taking time to respond to these questions. 

Where relevant, please indicate the specific sections of the NALS Background Questionnaire you are 
commenting on. The six sections are: 

Section A: General and Language Background 
Section B: Educational Background and Experiences 
Section C: Political and Social Participation 
Section D: Labor Force Participation 
Section E: Literacy Activities and Collaboration 
Section F: Demographic Information 

1 . ) What were the specific research questions in your study(ies) of the 1 992 NALS? 

2. ) How did the 1992 NALS Background Questionnaire items enhance or strengthen your ability to 

draw inferences and/or make policy recommendations? 

3. ) (a) How did the 1 992 NALS Background Questionnaire items limit your ability to draw inferences 

and/or make policy recommendations? b.) How were you able to rectify these problems? Please explain as 
precisely as possible. 

4. ) (a) What questions should be added to any or all of the sections of the 2002 National Assessment 

of Adult Literacy survey questionnaire, and (b) what existing items from the 1 992 NALS Background 
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Questionnaire should be revised and included in the 2002 survey? (c) How would these additions and 
revisions allow you or other researchers to make strong inferences from the obtained data? 

5. ) (a) What topic areas, aside from those indicated by the six Sections in the 1 992 NALS Background 

Questionnaire, do you think are important to examine for the 2002 National Assessment of Adult Literacy 
survey? (b) How would information on these topics be useful to you and other researchers and/or policy 
makers? 

6. ) Which questions and/or sections do you think are least useful and/or informative and should 

therefore be dropped from the 2002 National Assessment of Adult Literacy survey? 

7. ) Please list all papers and research reports you have published that pertain to your 

analyses of the 1992 NALS data. 

Thank you for your response to this survey. Your responses to the questions are, and will remain, 
confidential. 

Best Regards, 

M Cecil Smith 
Janet K. Sheehan-Holt 
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Listing of NCES Working Papers to Date 



Working papers can be downloaded as pdf files from the NCES Electronic Catalog 
(http://nces.ed.gov/pubsearch/) . You can also contact Sheilah Jupiter at (202) 502-7444 
(sheilahjupiter@ed.gov) if you are interested in any of the following papers. 

Listing of NCES Working Papers by Program Area 

No. Title NCES contact 



Baccalaureate and Beyond (B&B) 

98-15 Development of a Prototype System for Accessing Linked NCES Data 



Steven Kaufman 



Beginning Postsecondary Students (BPS) Longitudinal Study 

98-1 1 Beginning Postsecondary Students Longitudinal Study First Follow-up (BPS:96-98) Field 
Test Report 

98-1 5 Development of a Prototype System for Accessing Linked NCES Data 

1 999-1 5 Projected Postsecondary Outcomes of 1 992 High School Graduates 



Aurora D’Amico 

Steven Kaufman 
Aurora D’Amico 



Common Core of Data (CCD) 



95- 12 

96- 19 

97- 15 

97- 43 

98- 15 
1999-03 



Rural Education Data User’s Guide 
Assessment and Analysis of School-Level Expenditures 
Customer Service Survey: Common Core of Data Coordinators 
Measuring Inflation in Public School Costs 

Development of a Prototype System for Accessing Linked NCES Data 
Evaluation of the 1996-97 Nonfiscal Common Core of Data Surveys Data Collection, 
Processing, and Editing Cycle 



Decennial Census School District Project 

95- 1 2 Rural Education Data User’s Guide 

96- 04 Census Mapping Project/ School District Data Book 

98-07 Decennial Census School District Project Planning Report 



Samuel Peng 
William J. Fowler, Jr. 
Lee Hoffman 
William J. Fowler, Jr. 
Steven Kaufman 
Beth Young 



Samuel Peng 
Tai Phan 
Tai Phan 



Early Childhood Longitudinal Study (ECLS) 

96-08 How Accurate are Teacher Judgments of Students’ Academic Performance? Jerry West 

96- 1 8 Assessment of Social Competence, Adaptive Behaviors, and Approaches to Learning with Jerry West 

Young Children 

97- 24 Formulating a Design for the ECLS: A Review of Longitudinal Studies Jerry West 

97-36 Measuring the Quality of Program Environments in Head Start and Other Early Childhood Jerry West 

Programs: A Review and Recommendations for Future Research 

1 999- 0 1 A Birth Cohort Study: Conceptual and Design Considerations and Rationale Jerry West 

2000- 04 Selected Papers on Education Surveys: Papers Presented at the 1998 and 1 999 ASA and Dan Kasprzyk 

1 999 AAPOR Meetings 



Education Finance Statistics Center (EDFIN) 

94-05 Cost-of-Education Differentials Across the States 

96- 19 Assessment and Analysis of School-Level Expenditures 

97- 43 Measuring Inflation in Public School Costs 

98- 04 Geographic Variations in Public Schools’ Costs 

1 999- 1 6 Measuring Resources in Education: From Accounting to the Resource Cost Model 

Approach 



William J. Fowler, Jr. 
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High School and Beyond (HS&B) 
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1 999-05 Procedures Guide for Transcript Studies 
1 999-06 1 998 Revision of the Secondary School Taxonomy 



Samuel Peng 
Dawn Nelson 
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1 999-06 1 998 Revision of the Secondary School Taxonomy 



Dawn Nelson 
Dawn Nelson 
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No. 



Title 



NCES contact 



International Adult Literacy Survey (IALS) 

97-33 Adult Literacy: An International Perspective 

Integrated Postsecondary Education Data System (IPEDS) 

97- 27 Pilot Test of IPEDS Finance Survey 

98- 1 5 Development of a Prototype System for Accessing Linked NCES Data 

National Assessment of Adult Literacy (NAAL) 

98-17 Developing the National Assessment of Adult Literacy: Recommendations from 
Stakeholders 

1 999-09a 1 992 National Adult Literacy Survey: An Overview 

1 999-09b 1 992 National Adult Literacy Survey: Sample Design 

1 999-09c 1 992 National Adult Literacy Survey: Weighting and Population Estimates 

1 999-09d 1 992 National Adult Literacy Survey: Development of the Survey Instruments 

1 999-09e 1 992 National Adult Literacy Survey: Scaling and Proficiency Estimates 

1 999-09f 1 992 National Adult Literacy Survey: Interpreting the Adult Literacy Scales and Literacy 

Levels 

1 999- 09g 1 992 National Adult Literacy Survey: Literacy Levels and the Response Probability 

Convention 

2000- 05 Secondary Statistical Modeling With the National Assessment of Adult Literacy: 

Implications for the Design of the Background Questionnaire 
2000-06 Using Telephone and Mail Surveys as a Supplement or Alternative to Door-to-Door 
Surveys in the Assessment of Adult Literacy 

2000-07 “How Much Literacy is Enough?” Issues in Defining and Reporting Performance 
Standards for the National Assessment of Adult Literacy 
2000-08 Evaluation of the 1992 NALS Background Survey Questionnaire: An Analysis of Uses 
with Recommendations for Revisions 



National Assessment of Educational Progress (NAEP) 

95-12 Rural Education Data User’s Guide 

97-29 Can State Assessment Data be Used to Reduce State NAEP Sample Sizes? 

97-30 ACT’s NAEP Redesign Project: Assessment Design is the Key to Useful and Stable 
Assessment Results 

97-3 1 NAEP Reconfigured: An Integrated Redesign of the National Assessment of Educational 
Progress 

97-32 Innovative Solutions to Intractable Large Scale Assessment (Problem 2: Background 
Questionnaires) 

97-37 Optimal Rating Procedures and Methodology for NAEP Open-ended Items 

97- 44 Development of a SASS 1993-94 School-Level Student Achievement Subfile: Using 

State Assessments and State NAEP, Feasibility Study 

98- 1 5 Development of a Prototype System for Accessing Linked NCES Data 

1 999-05 Procedures Guide for Transcript Studies 

1 999-06 1 998 Revision of the Secondary School Taxonomy 



National Education Longitudinal Study of 1988 (NELS:88) 

95-04 National Education Longitudinal Study of 1 988: Second Follow-up Questionnaire Content 
Areas and Research Issues 

95-05 National Education Longitudinal Study of 1 988: Conducting Trend Analyses of NLS-72, 
HS&B, and NELS:88 Seniors 

95-06 National Education Longitudinal Study of 1 988: Conducting Cross-Cohort Comparisons 
Using HS&B, NAEP, and NELS:88 Academic Transcript Data 
95-07 National Education Longitudinal Study of 1 988: Conducting Trend Analyses HS&B and 
NELS:88 Sophomore Cohort Dropouts 
95-12 Rural Education Data User’s Guide 

95- 14 Empirical Evaluation of Social, Psychological, & Educational Construct Variables Used 

in NCES Surveys 

96- 03 National Education Longitudinal Study of 1 988 (NELS:88) Research Framework and 

Issues 



Marilyn Binkley 



Peter Stowe 
Steven Kaufman 



Sheida White 

Alex Sedlacek 
Alex Sedlacek 
Alex Sedlacek 
Alex Sedlacek 
Alex Sedlacek 
Alex Sedlacek 

Alex Sedlacek 

Sheida White 

Sheida White 

Sheida White 

Sheida White 



Samuel Peng 
Steven Gorman 
Steven Gorman 

Steven Gorman 

Steven Gorman 

Steven Gorman 
Michael Ross 

Steven Kaufman 
Dawn Nelson 
Dawn Nelson 



Jeffrey Owings 

Jeffrey Owings 

Jeffrey Owings 

Jeffrey Owings 

Samuel Peng 
Samuel Peng 

Jeffrey Owings 
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No. 



Title 



NCES contact 



98-06 

98-09 



98-15 

1999-05 

1999-06 

1999-15 



National Education Longitudinal Study of 1988 (NELS:88) Base Year through Second 
Follow-Up: Final Methodology Report 

High School Curriculum Structure: Effects on Coursetaking and Achievement in 

Mathematics for High School Graduates — An Examination of Data from the National 
Education Longitudinal Study of 1 988 
Development of a Prototype System for Accessing Linked NCES Data 
Procedures Guide for Transcript Studies 
1998 Revision of the Secondary School Taxonomy 
Projected Postsecondary Outcomes of 1992 High School Graduates 



National Household Education Survey (NHES) 

95- 12 Rural Education Data User’s Guide 

96- 13 Estimation of Response Bias in the NHES:95 Adult Education Survey 

96-14 The 1995 National Household Education Survey: Reinterview Results for the Adult 
Education Component 

96-20 1991 National Household Education Survey (NHES:91) Questionnaires: Screener, Early 

Childhood Education, and Adult Education 

96-2 1 1993 National Household Education Survey (NHES:93) Questionnaires: Screener, School 

Readiness, and School Safety and Discipline 

96-22 1995 National Household Education Survey (NHES:95) Questionnaires: Screener, Early 

Childhood Program Participation, and Adult Education 
96-29 Undercoverage Bias in Estimates of Characteristics of Adults and 0- to 2-Year-Olds in the 
1995 National Household Education Survey (NHES:95) 

96- 30 Comparison of Estimates from the 1995 National Household Education Survey 

(NHES:95) 

97- 02 Telephone Coverage Bias and Recorded Interviews in the 1993 National Household 

Education Survey (NHES: 93) 

97-03 1991 and 1995 National Household Education Survey Questionnaires: NHES:91 Screener, 

NHES:91 Adult Education, NHES:95 Basic Screener, and NHES:95 Adult Education 
97-04 Design, Data Collection, Monitoring, Interview Administration Time, and Data Editing in 
the 1993 National Household Education Survey (NHES:93) 

97-05 Unit and Item Response, Weighting, and Imputation Procedures in the 1993 National 
Household Education Survey (NHES:93) 

97-06 Unit and Item Response, Weighting, and Imputation Procedures in the 1995 National 
Household Education Survey (NHES:95) 

97-08 Design, Data Collection, Interview Timing, and Data Editing in the 1 995 National 
Household Education Survey 

97-19 National Household Education Survey of 1995: Adult Education Course Coding Manual 

97-20 National Household Education Survey of 1995: Adult Education Course Code Merge 

Files User’s Guide 

97-25 1 996 National Household Education Survey (NHES:96) Questionnaires: 

Screener/Household and Library, Parent and Family Involvement in Education and 
Civic Involvement, Youth Civic Involvement, and Adult Civic Involvement 
97-28 Comparison of Estimates in the 1996 National Household Education Survey 
97-34 Comparison of Estimates from the 1993 National Household Education Survey 
97-35 Design, Data Collection, Interview Administration Time, and Data Editing in the 1 996 
National Household Education Survey 

97-38 Reinterview Results for the Parent and Youth Components of the 1996 National 
Household Education Survey 

97-39 Undercoverage Bias in Estimates of Characteristics of Households and Adults in the 1 996 
National Household Education Survey 

97- 40 Unit and Item Response Rates, Weighting, and Imputation Procedures in the 1996 

National Household Education Survey 

98- 03 Adult Education in the 1990s: A Report on the 1991 National Household Education 

Survey 

98-10 Adult Education Participation Decisions and Barriers: Review of Conceptual Frameworks 
and Empirical Studies 



National Longitudinal Study of the High School Class of 1972 (NLS-72) 

95-12 Rural Education Data User’s Guide 



Ralph Lee 
Jeffrey Owings 

Steven Kaufman 
Dawn Nelson 
Dawn Nelson 
Aurora D’Amico 

Samuel Peng 
Steven Kaufman 
Steven Kaufman 

Kathryn Chandler 

Kathryn Chandler 

Kathryn Chandler 

Kathryn Chandler 

Kathryn Chandler 

Kathryn Chandler 

Kathryn Chandler 

Kathryn Chandler 

Kathryn Chandler 

Kathryn Chandler 

Kathryn Chandler 

Peter Stowe 
Peter Stowe 

Kathryn Chandler 

Kathryn Chandler 
Kathryn Chandler 
Kathryn Chandler 

Kathryn Chandler 

Kathryn Chandler 

Kathryn Chandler 

Peter Stowe 

Peter Stowe 

Samuel Peng 
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National Postsecondary Student Aid Study (NPSAS) 

96- 17 National Postsecondary Student Aid Study: 1996 Field Test Methodology Report 

National Study of Postsecondary Faculty (NSOPF) 

97- 26 Strategies for Improving Accuracy of Postsecondary Faculty Lists 

98- 15 Development of a Prototype System for Accessing Linked NCES Data 

2000-01 1999 National Study of Postsecondary Faculty (NSOPF :99) Field Test Report 



Private School Universe Survey (PSS) 

95-16 Intersurvey Consistency in NCES Private School Surveys 

95- 17 Estimates of Expenditures for Private K-12 Schools 

96- 16 Strategies for Collecting Finance Data from Private Schools 

96-26 Improving the Coverage of Private Elementary-Secondary Schools 

96- 27 Intersurvey Consistency in NCES Private School Surveys for 1993-94 

97- 07 The Determinants of Per-Pupil Expenditures in Private Elementary and Secondary 

Schools: An Exploratory Analysis 

97- 22 Collection of Private School Finance Data: Development of a Questionnaire 

98- 15 Development of a Prototype System for Accessing Linked NCES Data 

2000-04 Selected Papers on Education Surveys: Papers Presented at the 1998 and 1999 ASA and 
1999 AAPOR Meetings 

Recent College Graduates (RCG) 

98- 1 5 Development of a Prototype System for Accessing Linked NCES Data 



Schools and Staffing Survey (SASS) 

94-01 Schools and Staffing Survey (SASS) Papers Presented at Meetings of the American 
Statistical Association 

94-02 Generalized Variance Estimate for Schools and Staffing Survey (SASS) 

94-03 1991 Schools and Staffing Survey (SASS) Reinterview Response Variance Report 

94-04 The Accuracy of Teachers’ Self-reports on their Postsecondary Education: Teacher 

Transcript Study, Schools and Staffing Survey 

94- 06 Six Papers on Teachers from the 1990-91 Schools and Staffing Survey and Other Related 

Surveys 

95- 01 Schools and Staffing Survey: 1994 Papers Presented at the 1994 Meeting of the American 

Statistical Association 

95-02 QED Estimates of the 1990-91 Schools and Staffing Survey: Deriving and Comparing 
QED School Estimates with CCD Estimates 
95-03 Schools and Staffing Survey: 1990-91 SASS Cross-Questionnaire Analysis 
95-08 CCD Adjustment to the 1990-91 SASS: A Comparison of Estimates 
95-09 The Results of the 1993 Teacher List Validation Study (TLVS) 

95-10 The Results of the 1 991-92 Teacher Follow-up Survey (TFS) Reinterview and Extensive 

Reconciliation 

95-1 1 Measuring Instruction, Curriculum Content, and Instructional Resources: The Status of 
Recent Work 

95-12 Rural Education Data User’s Guide 

95-14 Empirical Evaluation of Social, Psychological, & Educational Construct Variables Used 
in NCES Surveys 

95-1 5 Classroom Instructional Processes: A Review of Existing Measurement Approaches and 
Their Applicability for the Teacher Follow-up Survey 
95-16 Intersurvey Consistency in NCES Private School Surveys 

95- 18 An Agenda for Research on Teachers and Schools: Revisiting NCES’ Schools and 

Staffing Survey 

96- 01 Methodological Issues in the Study of Teachers’ Careers: Critical Features of a Truly 

Longitudinal Study 

96-02 Schools and Staffing Survey (SASS): 1995 Selected papers presented at the 1995 Meeting 



of the American Statistical Association 

96-05 Cognitive Research on the Teacher Listing Form for the Schools and Staffing Survey 
96-06 The Schools and Staffing Survey (SASS) for 1 998-99: Design Recommendations to 
Inform Broad Education Policy 




Andrew G. Malizio 



Linda Zimbler 
Steven Kaufman 
Linda Zimbler 



Steven Kaufman 
Stephen Broughman 
Stephen Broughman 
Steven Kaufman 
Steven Kaufman 
Stephen Broughman 

Stephen Broughman 
Steven Kaufman 
Dan Kasprzyk 



Steven Kaufman 



Dan Kasprzyk 

Dan Kasprzyk 
Dan Kasprzyk 
Dan Kasprzyk 

Dan Kasprzyk 

Dan Kasprzyk 

Dan Kasprzyk 

Dan Kasprzyk 
Dan Kasprzyk 
Dan Kasprzyk 
Dan Kasprzyk 

Sharon Bobbitt & 
John Ralph 
Samuel Peng 
Samuel Peng 

Sharon Bobbitt 

Steven Kaufman 
Dan Kasprzyk 

Dan Kasprzyk 

Dan Kasprzyk 

Dan Kasprzyk 
Dan Kasprzyk 
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96-07 

96-09 

96-10 

96-11 

96-12 

96-15 

96-23 

96-24 

96-25 

96- 28 

97- 01 

97-07 

97-09 

97-10 

97-11 

97-12 

97-14 

97-18 

97-22 

97-23 

97-41 

97-42 

97- 44 

98- 01 
98-02 
98-04 
98-05 

98-08 

98-12 

98-13 

98-14 

98-15 

98-16 

1999-02 

1999-04 

1999-07 

1999-08 

1999-10 

1999-12 

1999-13 

1999-14 

1999-17 



Should SASS Measure Instructional Processes and Teacher Effectiveness? 

Making Data Relevant for Policy Discussions: Redesigning the School Administrator 
Questionnaire for the 1998-99 SASS 
1998-99 Schools and Staffing Survey: Issues Related to Survey Depth 
Towards an Organizational Database on America’s Schools: A Proposal for the Future of 
SASS, with comments on School Reform, Governance, and Finance 
Predictors of Retention, Transfer, and Attrition of Special and General Education 
Teachers: Data from the 1989 Teacher Followup Survey 
Nested Structures: District-Level Data in the Schools and Staffing Survey 
Linking Student Data to SASS: Why, When, How 
National Assessments of Teacher Quality 

Measures of Inservice Professional Development: Suggested Items for the 1998-1999 
Schools and Staffing Survey 

Student Learning, Teaching Quality, and Professional Development: Theoretical 
Linkages, Current Measurement, and Recommendations for Future Data Collection 
Selected Papers on Education Surveys: Papers Presented at the 1996 Meeting of the 
American Statistical Association 

The Determinants of Per-Pupil Expenditures in Private Elementary and Secondary 
Schools: An Exploratory Analysis 
Status of Data on Crime and Violence in Schools: Final Report 
Report of Cognitive Research on the Public and Private School Teacher Questionnaires 
for the Schools and Staffing Survey 1993-94 School Year 
International Comparisons of Inservice Professional Development 
Measuring School Reform: Recommendations for Future SASS Data Collection 
Optimal Choice of Periodicities for the Schools and Staffing Survey: Modeling and 
Analysis 

Improving the Mail Return Rates of SASS Surveys: A Review of the Literature 
Collection of Private School Finance Data: Development of a Questionnaire 
Further Cognitive Research on the Schools and Staffing Survey (SASS) Teacher Listing 
Form 

Selected Papers on the Schools and Staffing Survey: Papers Presented at the 1997 Meeting 
of the American Statistical Association 

Improving the Measurement of Staffing Resources at the School Level: The Development 
of Recommendations for NCES for the Schools and Staffing Survey (SASS) 
Development of a SASS 1993-94 School-Level Student Achievement Subfile: Using 
State Assessments and State NAEP, Feasibility Study 
Collection of Public School Expenditure Data: Development of a Questionnaire 
Response Variance in the 1993-94 Schools and Staffing Survey: A Reinterview Report 
Geographic Variations in Public Schools’ Costs 

SASS Documentation: 1993-94 SASS Student Sampling Problems; Solutions for 

Determining the Numerators for the SASS Private School (3B) Second-Stage Factors 
The Redesign of the Schools and Staffing Survey for 1999-2000: A Position Paper 
A Bootstrap Variance Estimator for Systematic PPS Sampling 
Response Variance in the 1994-95 Teacher Follow-up Survey 
Variance Estimation of Imputed Survey Data 

Development of a Prototype System for Accessing Linked NCES Data 
A Feasibility Study of Longitudinal Design for Schools and Staffing Survey 
Tracking Secondary Use of the Schools and Staffing Survey Data: Preliminary Results 
Measuring Teacher Qualifications 

Collection of Resource and Expenditure Data on the Schools and Staffing Survey 
Measuring Classroom Instructional Processes: Using Survey and Case Study Fieldtest 
Results to Improve Item Construction 
What Users Say About Schools and Staffing Survey Publications 
1993-94 Schools and Staffing Survey: Data File User’s Manual, Volume III: Public-Use 
Codebook 

1993- 94 Schools and Staffing Survey: Data File User’s Manual, Volume IV: Bureau of 
Indian Affairs (BIA) Restricted-Use Codebook 

1994- 95 Teacher Followup Survey: Data File User’s Manual, Restricted-Use Codebook 
Secondary Use of the Schools and Staffing Survey Data 



Dan Kasprzyk 
Dan Kasprzyk 

Dan Kasprzyk 
Dan Kasprzyk 

Dan Kasprzyk 

Dan Kasprzyk 
Dan Kasprzyk 
Dan Kasprzyk 
Dan Kasprzyk 

Mary Rollefson 

Dan Kasprzyk 

Stephen Broughman 

Lee Hoffman 
Dan Kasprzyk 

Dan Kasprzyk 
Mary Rollefson 
Steven Kaufman 

Steven Kaufman 
Stephen Broughman 
Dan Kasprzyk 

Steve Kaufman 

Mary Rollefson 

Michael Ross 

Stephen Broughman 
Steven Kaufman 
William J. Fowler, Jr. 
Steven Kaufman 

Dan Kasprzyk 
Steven Kaufman 
Steven Kaufman 
Steven Kaufman 
Steven Kaufman 
Stephen Broughman 
Dan Kasprzyk 
Dan Kasprzyk 
Stephen Broughman 
Dan Kasprzyk 

Dan Kasprzyk 
Kerry Gruber 

Kerry Gruber 

Kerry Gruber 
Susan Wiley 
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2000-04 Selected Papers on Education Surveys: Papers Presented at the 1998 and 1999 ASA and 
1999 AAPOR Meetings 




Dan Kasprzyk 
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Listing of NCES Working Papers by Subject 



No. Title 



NCES contact 



Adult education 

96-14 The 1 995 National Household Education Survey: Reinterview Results for the Adult 
Education Component 

96-20 1991 National Household Education Survey (NHES:91) Questionnaires: Screener, Early 

Childhood Education, and Adult Education 

96- 22 1 995 National Household Education Survey (NHES:95) Questionnaires: Screener, Early 

Childhood Program Participation, and Adult Education 
98-03 Adult Education in the 1990s: A Report on the 1991 National Household Education 
Survey 

98-1 0 Adult Education Participation Decisions and Barriers: Review of Conceptual Frameworks 
and Empirical Studies 

1 999-1 1 Data Sources on Lifelong Learning Available from the National Center for Education 
Statistics 

Adult literacy — see Literacy of adults 
American Indian - education 

1999-13 1993-94 Schools and Staffing Survey: Data File User's Manual, Volume IV: Bureau of 

Indian Affairs (BIA) Restricted-Use Codebook 

Assessment/achievement 
95-12 Rural Education Data User’s Guide 

95-13 Assessing Students with Disabilities and Limited English Proficiency 

97- 29 Can State Assessment Data be Used to Reduce State NAEP Sample Sizes? 

97-30 ACT’s NAEP Redesign Project: Assessment Design is the Key to Useful and Stable 
Assessment Results 

97-3 1 NAEP Reconfigured: An Integrated Redesign of the National Assessment of Educational 
Progress 

97-32 Innovative Solutions to Intractable Large Scale Assessment (Problem 2: Background 
Questions) 

97-37 Optimal Rating Procedures and Methodology for NAEP Open-ended Items 

97- 44 Development of a SASS 1993-94 School-Level Student Achievement Subfile: Using 

State Assessments and State NAEP, Feasibility Study 

98- 09 High School Curriculum Structure: Effects on Coursetaking and Achievement in 

Mathematics for High School Graduates — An Examination of Data from the National 
Education Longitudinal Study of 1 988 

Beginning students in postsecondary education 

98-1 1 Beginning Postsecondary Students Longitudinal Study First Follow-up (BPS:96-98) Field 
Test Report 

Civic participation 

97-25 1 996 National Household Education Survey (NHES:96) Questionnaires: 

Screener/Household and Library, Parent and Family Involvement in Education and 
Civic Involvement, Youth Civic Involvement, and Adult Civic Involvement 

Climate of schools 

95-14 Empirical Evaluation of Social, Psychological, & Educational Construct Variables Used 
in NCES Surveys 

Cost of education indices 

94- 05 Cost-of-Education Differentials Across the States 

Course-taking 

95- 12 Rural Education Data User’s Guide 



Steven Kaufman 
Kathryn Chandler 
Kathryn Chandler 
Peter Stowe 
Peter Stowe 
Lisa Hudson 



Kerry Gruber 



Samuel Peng 
James Houser 
Larry Ogle 
Larry Ogle 

Larry Ogle 

Larry Ogle 

Larry Ogle 
Michael Ross 

Jeffrey Owings 



Aurora D’Amico 



Kathryn Chandler 



Samuel Peng 



William J. Fowler, Jr. 



Samuel Peng 
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98-09 High School Curriculum Structure: Effects on Coursetaking and Achievement in 

Mathematics for High School Graduates— An Examination of Data from the National 
Education Longitudinal Study of 1988 
1999-05 Procedures Guide for Transcript Studies 
1999-06 1998 Revision of the Secondary School Taxonomy 

Crime 

97- 09 Status of Data on Crime and Violence in Schools: Final Report 

Curriculum 

95-1 1 Measuring Instruction, Curriculum Content, and Instructional Resources: The Status of 
Recent Work 

98- 09 High School Curriculum Structure: Effects on Coursetaking and Achievement in 

Mathematics for High School Graduates — An Examination of Data from the National 
Education Longitudinal Study of 1988 

Customer service 

1999- 10 What Users Say About Schools and Staffing Survey Publications 

2000- 02 Coordinating NCES Surveys: Options, Issues, Challenges, and Next Steps 

2000-04 Selected Papers on Education Surveys: Papers Presented at the 1998 and 1 999 ASA and 

1999 AAPOR Meetings 

Data quality 

97-13 Improving Data Quality in NCES: Database-to-Report Process 

Data warehouse 

2000-04 Selected Papers on Education Surveys: Papers Presented at the 1998 and 1 999 ASA and 
1999 AAPOR Meetings 



Design effects 

2000-03 Strengths and Limitations of Using SUDAAN, Stata, and WesVarPC for Computing 
Variances from NCES Data Sets 

Dropout rates, high school 

95- 07 National Education Longitudinal Study of 1988: Conducting Trend Analyses HS&B and 

NELS:88 Sophomore Cohort Dropouts 

Early childhood education 

96- 20 1991 National Household Education Survey (NHES:91) Questionnaires: Screener, Early 

Childhood Education, and Adult Education 

96- 22 1995 National Household Education Survey (NHES:95) Questionnaires: Screener, Early 

Childhood Program Participation, and Adult Education 

97- 24 Formulating a Design for the ECLS: A Review of Longitudinal Studies 

97- 36 Measuring the Quality of Program Environments in Head Start and Other Early Childhood 

Programs: A Review and Recommendations for Future Research 

1999- 01 A Birth Cohort Study: Conceptual and Design Considerations and Rationale 

Educational attainment 

98- 1 1 Beginning Postsecondary Students Longitudinal Study First Follow-up (BPS:96-98) Field 

Test Report 

Educational research 

2000- 02 Coordinating NCES Surveys: Options, Issues, Challenges, and Next Steps 

Employment 

96-03 National Education Longitudinal Study of 1988 (NELS:88) Research Framework and 
Issues 

98-1 1 Beginning Postsecondary Students Longitudinal Study First Follow-up (BPS: 96-98) Field 
Test Report 



Jeffrey Owings 



Dawn Nelson 
Dawn Nelson 



Lee Hoffman 



Sharon Bobbitt & 
John Ralph 
Jeffrey Owings 



Dan Kasprzyk 
Valena Plisko 
Dan Kasprzyk 



Susan Ahmed 



Dan Kasprzyk 



Ralph Lee 



Jeffrey Owings 



Kathryn Chandler 

Kathryn Chandler 

Jerry West 
Jerry West 

Jerry West 
Aurora D’Amico 



Valena Plisko 



Jeffrey Owings 
Aurora D’Amico 
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Faculty - higher education 

97- 26 Strategies for Improving Accuracy of Postsecondary Faculty Lists 

2000-01 1999 National Study of Postsecondary Faculty (NSOPF:99) Field Test Report 

Finance - elementary and secondary schools 

94- 05 Cost-of-Education Differentials Across the States 

96- 1 9 Assessment and Analysis of School-Level Expenditures 

98- 01 Collection of Public School Expenditure Data: Development of a Questionnaire 

1 999-07 Collection of Resource and Expenditure Data on the Schools and Staffing Survey 

1 999-16 Measuring Resources in Education: From Accounting to the Resource Cost Model 

Approach 

Finance - postsecondary 

97- 27 Pilot Test of IPEDS Finance Survey 

Finance - private schools 

95- 1 7 Estimates of Expenditures for Private K- 1 2 Schools 

96- 16 Strategies for Collecting Finance Data from Private Schools 

97- 07 The Determinants of Per-Pupil Expenditures in Private Elementary and Secondary 

Schools: An Exploratory Analysis 

97- 22 Collection of Private School Finance Data: Development of a Questionnaire 

1 999- 07 Collection of Resource and Expenditure Data on the Schools and Staffing Survey 

Geography 

98- 04 Geographic Variations in Public Schools’ Costs 

Imputation 

2000- 04 Selected Papers on Education Surveys: Papers Presented at the 1998 and 1 999 ASA and 

1 999 AAPOR Meetings 

Inflation 

97-43 Measuring Inflation in Public School Costs 

Institution data 

2000-0 1 1 999 National Study of Postsecondary Faculty (NSOPF:99) Field Test Report 

Instructional resources and practices 

95-1 1 Measuring Instruction, Curriculum Content, and Instructional Resources: The Status of 
Recent Work 

1999-08 Measuring Classroom Instructional Processes: Using Survey and Case Study Field Test 
Results to Improve Item Construction 



International comparisons 

97-1 1 International Comparisons of Inservice Professional Development 
97-16 International Education Expenditure Comparability Study: Final Report, Volume I 

97-17 International Education Expenditure Comparability Study: Final Report, Volume II, 

Quantitative Analysis of Expenditure Comparability 



Libraries 

94-07 Data Comparability and Public Policy: New Interest in Public Library Data Papers 
Presented at Meetings of the American Statistical Association 
97-25 1 996 National Household Education Survey (NHES:96) Questionnaires: 

Screener/Household and Library, Parent and Family Involvement in Education and 
Civic Involvement, Youth Civic Involvement, and Adult Civic Involvement 



Limited English Proficiency 

95-13 Assessing Students with Disabilities and Limited English Proficiency 



Linda Zimbler 
Linda Zimbler 



William J. Fowler, Jr. 
William J. Fowler, Jr. 
Stephen Broughman 
Stephen Broughman 
William J. Fowler, Jr. 



Peter Stowe 



Stephen Broughman 
Stephen Broughman 
Stephen Broughman 

Stephen Broughman 
Stephen Broughman 



William J. Fowler, Jr. 



Dan Kasprzyk 



William J. Fowler, Jr. 



Linda Zimbler 



Sharon Bobbitt & 
John Ralph 
Dan Kasprzyk 



Dan Kasprzyk 
Shelley Bums 
Shelley Bums 



Carrol Kindel 
Kathryn Chandler 



James Houser 
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Literacy of adults 

98-17 Developing the National Assessment of Adult Literacy: Recommendations from 
Stakeholders 

1 999-09a 1 992 National Adult Literacy Survey: An Overview 

1 999-09b 1 992 National Adult Literacy Survey: Sample Design 

1 999-09c 1 992 National Adult Literacy Survey: Weighting and Population Estimates 

1 999-09d 1 992 National Adult Literacy Survey: Development of the Survey Instruments 

1 999-09e 1 992 National Adult Literacy Survey: Scaling and Proficiency Estimates 

1 999-09f 1 992 National Adult Literacy Survey: Interpreting the Adult Literacy Scales and Literacy 

Levels 

1 999-09g 1 992 National Adult Literacy Survey: Literacy Levels and the Response Probability 

Convention 

1 999- 1 1 Data Sources on Lifelong Learning Available from the National Center for Education 

Statistics 

2000- 05 Secondary Statistical Modeling With the National Assessment of Adult Literacy: 

Implications for the Design of the Background Questionnaire 
2000-06 Using Telephone and Mail Surveys as a Supplement or Alternative to Door-to-Door 
Surveys in the Assessment of Adult Literacy 

2000-07 “How Much Literacy is Enough?” Issues in Defining and Reporting Performance 
Standards for the National Assessment of Adult Literacy 
2000-08 Evaluation of the 1 992 NALS Background Survey Questionnaire: An Analysis of Uses 
with Recommendations for Revisions 

Literacy of adults - international 

97- 33 Adult Literacy: An International Perspective 

Mathematics 

98- 09 High School Curriculum Structure: Effects on Coursetaking and Achievement in 

Mathematics for High School Graduates — An Examination of Data from the National 
Education Longitudinal Study of 1988 

1999-08 Measuring Classroom Instructional Processes: Using Survey and Case Study Field Test 
Results to Improve Item Construction 

Parental involvement in education 

96- 03 National Education Longitudinal Study of 1 988 (NELS:88) Research Framework and 

Issues 

97- 25 1 996 National Household Education Survey (NHES:96) Questionnaires: 

Screener/Household and Library, Parent and Family Involvement in Education and 
Civic Involvement, Youth Civic Involvement, and Adult Civic Involvement 
1 999-01 A Birth Cohort Study: Conceptual and Design Considerations and Rationale 

Participation rates 

98- 10 Adult Education Participation Decisions and Barriers: Review of Conceptual Frameworks 

and Empirical Studies 

Postsecondary education 

1 999-1 1 Data Sources on Lifelong Learning Available from the National Center for Education 
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